diff --git a/.gitignore b/.gitignore index 3a6094ba..c774b2c5 100644 --- a/.gitignore +++ b/.gitignore @@ -8,7 +8,7 @@ sequencer.db-shm sequencer.db-wal /out/ examples/canonical-app/out/ -/.DS_Store +.DS_Store .vscode/ soljson-latest.js **/states/ diff --git a/AGENTS.md b/AGENTS.md index 9382fc8c..efddca46 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,6 +29,9 @@ In order of importance: - **App-specific sequencer.** The sequencer may link against the application, enabling validation and execution at ingress time. This is a deliberate design choice. - **Soft confirmations may be invalidated.** Under adversarial conditions (network, infrastructure, provider, or L1 outages), soft confirmations can be rolled back via recovery. This is by design, not a bug — it is what makes the sequencer sound in the face of liveness failures. - **App UX may depend on the sequencer.** Without the sequencer, user experience may degrade substantially. This is an acceptable tradeoff: the on-chain scheduler remains the canonical source of truth; the sequencer only accelerates the UX. +- **SQLite-centered local coordination.** Components publish and consume durable local facts through their owned SQLite tables. The on-chain scheduler remains canonical authority; SQLite is the sequencer's local coordination plane. HTTP ingress ↔ inclusion lane MPSC/oneshot is the deliberate exception because low-latency request/response over the lane's in-memory application is unwieldy through SQLite. Do not turn that exception into a general in-memory component bus. +- **Assumption-driven robustness.** Every hardening mechanism must name the invariant it protects, the assumptions under which it is needed, and a trigger for revisiting them. Machinery for failures outside the supported model enlarges the state surface developers must audit and can make the system less robust rather than more. +- **The complexity budget belongs to concurrency, mutual exclusion, durability, and hostile-L1 robustness.** This sequencer is not algorithmically complex. A large file is a smell: an invariant we're not seeing, a reasonable assumption we're not taking, or plain over-engineering. Judge every mechanism — current or proposed — against its weight. ## Sequencer / Scheduler Duality @@ -68,11 +71,11 @@ Scheduler-acceptance semantics exist in exactly three implementations that must 2. the off-chain acceptance predicate — `ProtocolTiming::scheduler_accepts` ([`sequencer-core/src/protocol.rs`](sequencer-core/src/protocol.rs)), which feeds `safe_accepted_batches`; 3. the inclusion lane's live prediction (drain + execution order). -The expected-nonce fold is homed next to `scheduler_accepts` as `advance_expected_batch_nonce` (same file); the submitter's `decide_submit_start` consumes it, and `populate_safe_accepted_batches` keeps a deliberate inline copy (its advance is interleaved with storage-only side effects — the R2 content-identity check and the divergence freeze — that can't move below the protocol layer). Touching any of these means re-checking the others — their agreement is the system's most load-bearing invariant (see [`docs/invariants.md`](docs/invariants.md)). +The expected-nonce fold is homed next to `scheduler_accepts` as `advance_expected_batch_nonce` (same file); the submitter's `decide_submit_start` consumes it, and `populate_safe_accepted_batches` keeps a deliberate inline copy (its advance is interleaved with storage-only side effects — the content-identity check and the divergence freeze — that can't move below the protocol layer). Touching any of these means re-checking the others — their agreement is the system's most load-bearing invariant (see [`docs/invariants.md`](docs/invariants.md)). Two mechanical facts the agreement rests on: -- **Drain attribution.** At a safe-frontier advance, the newly-drained directs are sequenced into the **new** frame — the frame stamped with the **new** `safe_block`. So frame K's wire content reads "directs ≤ S_K, then user ops validated on top of them", exactly the scheduler's drain-before-ops rule. +- **Drain attribution.** At an eligible five-safe-block clock advance, every accumulated newly-safe direct is sequenced into the **new** frame — the frame stamped with the latest observed `safe_block`. So frame K's wire content reads "directs ≤ S_K, then user ops validated on top of them", exactly the scheduler's drain-before-ops rule. A clock tick may have an empty direct prefix. - **Empty batches are never stale and consume the nonce** (no first frame to measure staleness against). Consistent across all implementations, test-pinned. ## Batch Staleness and Recovery @@ -96,14 +99,18 @@ Rather than waiting for a batch to go stale on L1, the sequencer uses a **danger The cycle crosses a process boundary by design: the in-process [`DangerDetector`](sequencer/src/recovery/detector.rs) polls -`Storage::check_danger` on a cadence and **exits the process** when any -non-`Safe` arm fires (stopping the process is how the sequencer goes offline); -the orchestrator respawns; startup syncs the L1 safe head, re-runs -`check_danger`, and [`decide_startup_action`](sequencer/src/recovery/mod.rs) -dispatches — `Proceed` (no recovery writes), `RecoverTip` (invalidate the aging -Tip directly; it has no L1 footprint, so no flush), `FlushAndCascade` (flush -every wallet-nonce slot, re-sync, cascade everything past the gold frontier), -or `Refuse` (surface to the operator). Then normal operation resumes. +`Storage::check_danger` on a cadence and returns a non-`Safe` worker exit; the +runtime closes intake and drains before the command returns non-zero (stopping +the process is how the sequencer goes offline). Terminal containment has the +two-second hard abort fallback; expected-recovery/retryable exits are graceful; +the orchestrator respawns; on every boot the startup +reducer inspects local facts before the first provider call, executes at most +one phase, and re-inspects after every completed phase. Closed recovery is +structurally `Flush → inspect → Sync → inspect → Cascade → inspect`, with an +ephemeral safe-block witness carried only by that boot attempt. A clean +decision permits task-free runtime preparation; the same reducer is invoked +again over one consistent fact set before the single-use `RuntimeAdmission` +witness is minted, then worker launch is infallible and non-yielding. The authoritative dispatch table, the "everything past gold is doomed" model, and the per-path rationale live in @@ -119,7 +126,11 @@ When the sequencer's view of L1 stops advancing — most often because the RPC g ### Formal verification -The preemptive recovery design is verified by bounded TLA+ model checking. See [`docs/recovery/`](docs/recovery/) for the full design, TLA+ specs, and design history. When touching recovery code, read the TLA+ first. +The recovery design is verified by complementary bounded TLA+ models: +`preemptive.tla` for slot/batch safety and `admission.tla` for startup phase +ordering and runtime admission. See [`docs/recovery/`](docs/recovery/) for the +full design, specs, and history. When touching recovery code, read both current +models first. ## Threat Model (brief) @@ -149,9 +160,9 @@ Top-level layout follows the system's data flow. Each sequencer module correspon ### Sequencer module layout - `sequencer/src/lib.rs` — public sequencer API. The thin binary entrypoints live in `examples/wallet-sequencer/`. -- `sequencer/src/harness.rs` — CLI harness: the `setup`/`run`/`flush-mempool` subcommand parser, `dispatch`, and the R4 exit-code projection. An app's `main` is ~5 lines (`run_main` + a genesis-app closure). +- `sequencer/src/harness.rs` — CLI harness: the `setup`/`run`/`flush-mempool` subcommand parser, `dispatch`, and the exit-code projection. An app's `main` is ~5 lines (`run_main` + a genesis-app closure). - `sequencer/src/http.rs` — shared HTTP error type, JSON `ErrorResponse`, `ApiConfig`, and `axum::serve` orchestration. -- `sequencer/src/runtime/` — process orchestration: `setup` (phase A — pin identity, initial sync, genesis snapshot, `setup_complete` marker), `run` (phase B — boot workers from a set-up DB), `flush` (`flush-mempool`), plus `config`, `error` (incl. exit-code projection), `shutdown`, shared `clock::unix_now_ms`, and the `workers` lifecycle. +- `sequencer/src/commands/` — the operator command brackets: `setup` (phase A — pin identity, initial sync, genesis snapshot, atomic `setup_complete` fact), `run` (phase B — recover, prepare, admit, and boot workers; its `workers` supervisor lives beside it), and `flush` (`flush-mempool`). `sequencer/src/commands/` also owns the command-scoped `config` and `error` taxonomy (incl. the exit-code projection); `sequencer/src/runtime/` is exactly the runtime authority capabilities — the process lock and `shutdown` (runtime scope/containment) — consumed crate-wide. `L1Config` lives in `sequencer/src/l1/`; the crate-wide wall clock is `sequencer/src/clock.rs`. - `sequencer/src/ingress/` — public write path. - `api.rs` — `POST /tx` handler, JSON-rejection mapping. - `inclusion_lane/` — single-lane hot-path loop (`mod.rs`), catch-up replay, config, error types. @@ -166,21 +177,42 @@ Top-level layout follows the system's data flow. Each sequencer module correspon - `provider.rs` — alloy provider construction. - `partition.rs` — long-block-range retry helper. - `sequencer/src/recovery/` — preemptive recovery startup procedure (`mod.rs`), runtime danger detector (`detector.rs`), and mempool flusher (`flusher.rs`). -- `sequencer/src/storage/` — SQLite persistence, split by writer role (`ingress`, `egress`, `l1_inputs`, `l1_submission`, `recovery`, `admin`, `safe_accepted_batches`, `snapshot_dumps`, plus shared `mod`, `open`, `convert`, `queries`, `mutations`, and `migrations/`). +- `sequencer/src/storage/` — SQLite persistence, split by writer role (`ingress`, `egress`, `l1_inputs`, `l1_submission`, `recovery`, `admin`, `safe_accepted_batches`, `snapshot_dumps`, plus shared `history`, `mod`, `open`, `convert`, `queries`, `mutations`, and `migrations/`). ## Key Concepts - **Chunk** — bounded list of user ops processed and persisted together to amortize SQLite cost. - **Frame** — ordering boundary; commits `safe_block` + user ops. - **Batch** — list of frames posted on-chain as one L1 transaction (SSZ-encoded). -- **Inclusion lane** — hot-path single-lane loop that dequeues, executes, persists, and rotates frame/batch boundaries. The only writer of open batch/frame state. +- **Inclusion lane** — single ordering lane with two regimes: a latency-critical user-op regime that commits at most one bounded chunk per fast turn, and an L1-reconciliation regime that advances logical frame time to the latest observed safe head once at least five safe blocks have accumulated. That slow turn executes every accumulated direct input and promotes snapshots, even when the new frame has no directs. The lane is the only writer of open batch/frame state and the system's execution bottleneck. - **Batch submitter** — stateless worker that bulk-submits all pending batches each tick. Nonces are assigned by storage (structural `parent.nonce + 1`) when batches are closed; the submitter just reads them. - **Danger detector** — background worker that polls `Storage::check_danger` on a fixed cadence and exits with `RecoveryRequired` when any non-`Safe` danger status fires. Never writes to the DB; never talks to L1. Crashes the process so startup recovery or refusal can run. -- **Fee oracle** — setup pins either a fixed exponent or a reviewed Uniswap V3 WETH/X TWAP tuple into deployment identity, and writes the first `log_gas_price` (+ freshness stamp) in both modes. Fixed mode has no worker; Uniswap refreshes `batch_policy.log_gas_price` on a poll loop. Transient L1 failures at `run` boot and at runtime retain the persisted price until `log_gas_price_updated_at_ms` exceeds the L1 read-staleness window; misconfig stays terminal. The 10× margin lives in `batch_policy.log_slack`; frame fees stay immutable until the next frame opens. -- **Input reader** — ingests safe inputs from L1 InputBox into SQLite. -- **L2 tx feed** — DB-backed ordered-tx stream used by WS subscribers. +- **Fee oracle** — setup pins either a fixed exponent or a reviewed Uniswap V3 WETH/X TWAP tuple into deployment identity, and writes the first `log_gas_price` (+ observation stamp) in both modes. Setup requires a successful live quote; `run` performs no fee-source read before recovery/admission. Fixed mode has no worker; Uniswap launches a lazy refresher that immediately attempts a quote, persists successes, and retains the last price while logging and retrying transient source failures. The stamp is telemetry, not a runtime-admission or expiry gate. A shared-endpoint outage/stale view is already detected from L1 safe-head progress; a fee-source-only outage is an accepted economic residual (stale-low may subsidize DA, stale-high may reject users), not a canonical-correctness fault. Deterministic source misconfiguration, fatal arithmetic, and persistent storage faults remain terminal. The 10× margin lives in `batch_policy.log_slack`; it is a buffer rather than a bound on market movement, and frame fees stay immutable until the next frame opens. +- **Input reader** — ingests safe inputs from L1 InputBox and atomically maintains the durable safe head, accepted-batch projection, and content-identity divergence marker in SQLite. It does not hand an in-memory cursor to the inclusion lane. +- **L2 tx feed** — DB-backed ordered-tx stream used by WS subscribers. The + existing endpoint still paginates by the physical SQLite rowid cursor. + SQLite now also stores the canonical `ExecutedInputCount` attribution for + every application input; switching the public feed and history-version + handshake to that coordinate remains Track 3 API work. +- **Application progress** — scheduler-owned + `(ExecutedInputCount, last_executed_safe_block)` embedded in every + application dump. Shared execution functions advance it and return the + input's pre-execution offset. SQLite records that offset atomically with the + corresponding valid replay row; only the WebSocket/HTTP projection remains + Track 3 work. +- **History version** — `(EraId, RecoveryGeneration)`. The durable metadata + foundation is landed: a new baseline mints an immutable UUIDv4 era and starts + generation zero; standard recovery increments it exactly once iff its + transaction invalidates at least one valid batch. The current feed does not + expose or enforce the pair yet. - **Soft confirmation** — sequencer's predicted ordering, emitted before the batch lands on L1. -- **Snapshot** — durable copy of the app's canonical state at a known L2-tx offset; *pending* at batch close, *promoted* to finalized on L1 observation (per-range, atomically with the drain), garbage-collected when superseded. Backs catch-up, the watchdog, and indexers. Lifecycle + rationale (incl. the promote/drain crash-safety): [`docs/snapshots/lifecycle.md`](docs/snapshots/lifecycle.md). +- **Snapshot** — durable copy of the app's canonical state at one physical + replay cursor and one canonical `ExecutedInputCount`; *pending* at batch + close, *promoted* to finalized on L1 observation (per-range, atomically with + the drain), garbage-collected when superseded. Catch-up refuses if the + loaded app count, stored snapshot count, or per-row execution attributions + disagree. Lifecycle + rationale (incl. the promote/drain crash-safety): + [`docs/snapshots/lifecycle.md`](docs/snapshots/lifecycle.md). ## Domain Truths @@ -188,7 +220,7 @@ Top-level layout follows the system's data flow. Each sequencer module correspon - **Deposits are direct-input-only** (L1 → L2) and must not be represented as user ops. - Rejections (`InvalidNonce`, `InvalidMaxFee`, `InsufficientFeeBalance`) produce no state mutation and are not persisted. These are protocol-level rejection semantics every app must implement: nonces prevent user-op replay, fees prevent spam against the sequencer's DA budget. ("Fee", not "gas" — the fee tracks DA; compute metering, if it ever exists, is a separate future concept.) - Included txs are persisted as frame/batch data in `batches`, `frames`, `user_ops`, `safe_inputs`, and `sequenced_l2_txs`. Recovery metadata lives in `safe_accepted_batches`; batch lifecycle state (sealed/invalidated) lives on the `batches` row itself as write-once timestamps. -- Frame fee is persisted in `frames.fee` and is fixed for the lifetime of that frame. The next frame's fee is sampled from `batch_policy_derived.recommended_fee` at rotation; oracle bootstrap writes the price before any Tip can sample it, and `log_slack` applies the 10× margin in log space. +- Frame fee is persisted in `frames.fee` and is fixed for the lifetime of that frame. The next frame's fee is currently sampled from `batch_policy_derived.recommended_fee` at rotation; oracle bootstrap writes the price before any Tip can sample it, and `log_slack` applies the 10× margin in log space. This is present behavior, not a reason for the five-block clock policy; hoisting fee to the batch is a later design with its own trade-offs. - Wallet state (balances, nonces) is in-memory today — not persisted. - **EIP-712 domain fields:** `name`, `version`, `chainId`, `verifyingContract`. `chainId` and `verifyingContract` come from `CARTESI_SEQUENCER_BLOCKCHAIN_ID` and `CARTESI_SEQUENCER_APP_ADDRESS` (validated against the RPC chain id at startup). All four fields must be present on both sides — both the sequencer and the on-chain scheduler construct the domain via `sequencer_core::build_input_domain`, the canonical shared constructor. @@ -198,34 +230,42 @@ Top-level layout follows the system's data flow. Each sequencer module correspon - **Classification is by sender address**, not by a tag byte: - Sender == batch-submitter address → SSZ-decoded as `Batch` (scheduler side). The sequencer does not ingest its own batch submissions as direct inputs. - Any other sender → stored verbatim as a direct input (deposit). -- The payload is opaque to the classification layer. Application-specific decoding happens inside `Application::execute_direct_input`. +- The payload is opaque to the classification layer. Application-specific decoding happens inside `Application::apply_direct_input`, reached only through the shared `execute_direct_input` boundary. ## Application Trait Contract -Implementors of the `Application` trait must respect these contracts. The sequencer assumes them without runtime enforcement. The full, code-grounded contract — method table, dump round-trip durability, the safe-block clock — is **owned by [`docs/protocol/application-contract.md`](docs/protocol/application-contract.md)**; the essentials follow. +Implementors of the `Application` trait must respect these contracts. The shared execution boundary enforces scheduler-owned count/clock progress; application-specific determinism and mutation remain self-trusted. The full, code-grounded contract — method table, dump round-trip durability, the safe-block clock — is **owned by [`docs/protocol/application-contract.md`](docs/protocol/application-contract.md)**; the essentials follow. ### Replay determinism The sequencer persists every included user op and every ingested direct input. On restart, catch-up replays them in order against a fresh `Application` instance to rebuild state. **Any input that succeeded live must succeed on replay.** -- `execute_direct_input` and `execute_valid_user_op` must not return `AppError::Internal` for any byte sequence that previously executed successfully. Catch-up treats `Internal` as fatal: it aborts startup and leaves the sequencer unable to resume. +- `apply_direct_input` and `apply_valid_user_op` must not return `AppError::Internal` for any byte sequence that previously executed successfully. The canonical scheduler, catch-up, and recovery fold treat `Internal` as fatal: no canonical successor is defined. - Prefer `ExecutionOutcome::Invalid` for malformed or ill-typed input caught at the app level. Reserve `AppError::Internal` for genuine invariant violations ("validated user op cannot pay fee") — real bugs, not adversarial inputs. `Invalid` is replay-safe; `Internal` is not. - `validate_user_op` must be pure over the current app state. No side effects, no time dependence, no randomness. ### No implicit state -Application state changes must flow exclusively through `execute_valid_user_op` and `execute_direct_input`. Mutating state from `validate_user_op` breaks replay determinism. +Application-specific state changes flow exclusively through the `apply_valid_user_op` and `apply_direct_input` hooks. Scheduler-owned `ApplicationProgress` (executed-input count plus safe-block clock) changes only through the shared free execution functions. Mutating state from `validate_user_op` breaks replay determinism. ### One execution entry point -User ops are executed only through `sequencer_core::application::validate_and_execute_user_op` (a free function, deliberately not an overridable trait method): it enforces the protocol-level `max_fee >= current_fee` guard before app validation, so no `Application` impl can skip it. Both the inclusion lane and the canonical scheduler call it — part of the duality agreement. +User ops are executed only through `sequencer_core::application::validate_and_execute_user_op`; already-validated user ops and directs use the shared `execute_valid_user_op` / `execute_direct_input` free functions. Raw hooks and mutable progress access require distinct borrowed opaque capabilities whose constructors are private to this boundary. It preflights the checked successor, verifies progress stayed unchanged after validation and after the hook on both `Ok` and `Err`, commits only after `Ok`, then re-reads the getter to assert accessor coherence. Count zero implies clock zero. `AppError` is fatal and defines no canonical successor; callers discard the application instance rather than resume it. The inclusion lane, canonical scheduler, catch-up, and recovery fold all use this boundary — part of the duality agreement. ## Hot-Path Invariants -- API ack is tied to chunk durability, not frame/batch closure. "Durable" means power-loss-durable: WAL with `synchronous=FULL`, so every commit fsyncs before anything externalizes on it (review R3). -- Chunk commit and ack remain low-latency; frame closure is orthogonal and can happen less frequently. +- API ack is tied to chunk durability, not frame/batch closure. "Durable" means power-loss-durable: WAL with `synchronous=FULL`, so every commit fsyncs before anything externalizes on it. +- Command admission is fact-derived: the kernel process lock excludes concurrent owners, `setup_complete` orders commands two-sided, and `canonical_divergence` is the one absorbing refusal (cockroach rebuild only). There is no lifecycle admission state machine and no operator acknowledgement — standard recovery is automatic, and restart policy after a terminal fault is the exit-code contract (30 = do not restart, page). The only durable telemetry is the `terminal_faults` black box: append-only terminal-cause rows, written best-effort and verdict-neutrally — telemetry never changes a command's verdict, and nothing reads the black box for decisions. `run` admits only after fallible preparation, by re-running the reducer over one consistent fact set immediately before non-yielding worker launch; SIGKILL/OOM/cancellation writes nothing (the next boot proceeds and re-derives from facts). In-process terminal containment sets the bit, arms an independent two-second process-abort watchdog, requests cooperative shutdown, and only then appends the black box's terminal-cause row (best-effort telemetry). The watchdog holds a weak process-lock witness and aborts only if a controller, worker, or nested blocking task has not drained by the deadline; ordinary operator/recovery shutdown stays graceful. If the terminal-cause write fails, the exit code and logs still carry the verdict, and a persistent fault re-detects fail-loud on the next boot that reads it. One process per data dir is kernel-enforced by an exclusive lock (`sequencer/src/runtime/process_lock.rs`); the controller retains it through black-box settlement and nested work retains clones until it actually stops. Cleanup polls all workers concurrently so a hung drain cannot hide a terminal exit. An already-authorized effect may complete, and snapshot streams check containment only at start (they are non-authority-bearing immutable reads; per-chunk stream cancellation is not a containment guarantee). +- Setup/rebuild, normal-run recovery, and maintenance flush deliberately retain distinct typed controllers; do not combine their unrelated facts into a generic command state machine. `flush-mempool` is flush-only and requires a completed setup and no divergence — there is no admission state for it to restore or erase, and a successful wallet flush is never treated as proof the runtime is clean. +- Initial setup/rebuild creates the baseline schema, UUIDv4 `EraId`, and generation zero in one transaction. A rebuild leaves `base_executed_input_count` and `base_safe_input_index` NULL until cockroach fill derives `K = recovered_app.executed_input_count()` and the recovery root's exclusive safe-input cursor, then binds both atomically with the initial finalized snapshot; setup completion requires both non-NULL bases and the snapshot. `K` is application history, not physical `l2_tx_index` cursor padding. The safe-input base is a durable drain floor: standard recovery derives the next cursor as `max(base_safe_input_index, max valid attribution + 1)`, so invalidating the cockroach root cannot re-execute inputs already represented by `S'`. Cockroach recovery deliberately remains an explicit fresh/wiped-directory operator flow—no automated DB replacement, clone detection, distributed fencing, or partial-fill resume state machine. A retained early incomplete DB reuses its unexposed era; a fail-loud partial fill requires wipe/retry and therefore a new unexposed era. +- Chunk commit and ack remain low-latency; frame closure is orthogonal and can happen less frequently. The product contract is under 500 ms; same-host release sweeps across the authority cutover found ACK p99 at or below ~50 ms through concurrency 256 with zero rejections, and concurrency-1 HTTP ACK p50 around 13 ms while submit-to-matching-WS-event p50 was roughly double — always name which metric "round-trip" means. Same-host numbers are method-specific regression evidence, never portable capacity claims (the load clients contend with the sequencer at high concurrency). +- Never add a divergence-marker query, provider call, or reader mailbox to every user-op chunk. The content-identity check is complete for at/above-anchor, simulated-accepted batch content identity, not arbitrary canonical/application divergence (its scope boundary is [I9](docs/invariants.md)). The danger detector owns prompt process-wide reaction; the lane's existing time-gated frontier read opportunistically returns a typed divergence instead of a usable frontier. No extra poll or reaction deadline exists. +- Entry to reconciliation is bounded by dequeued attempts, not only included bytes: rejected requests do not advance the batch target. One existing `max_user_ops_per_chunk` dequeue chunk is the fast-turn boundary before the outer frontier check. This adds no timer, cursor, or fairness knob and prevents sustained rejected traffic from starving the slow turn. Returning to the outer loop normally performs only cheap time-gate bookkeeping; it does not fsync or query SQLite after every chunk. +- Once reconciliation starts, process the complete accumulated newly-safe range, including supported catch-up/backlog conditions, before returning to user ops. Supported applications are assumed to digest that range promptly; paging bounds scratch memory/read-query size, not the atomic drain transaction or logical work. Add preemption or a durable partial cursor only if production measurements disprove that assumption. +- During an admitted live run, L1 reconciliation has no tight acknowledgement SLA. Its semantic clock trigger is block-only: if the latest persisted safe head `H` is at least five blocks beyond the open frame's `safe_block` `S`, reconcile once and open exactly one frame at `H`. Never synthesize intermediary frames after a jump; `H` becomes the new anchor. Bootstrap and recovery instead anchor a fresh Tip at their proven checkpoint/current safe head; they are not live clock ticks. The time gate controls SQLite observation load, not clock semantics. Direct-input presence is work discovered at the turn, never another trigger. +- A newly-safe direct may therefore wait below the five-block threshold. When the tick arrives it still executes with its exact inclusion-block clock, before user ops at the new frame clock. Five blocks is a sequencer policy chosen comfortably inside the happy-path deposit budget; if actual safe-head publication cadence threatens that budget, revisit the constant from measurements rather than adding interpolation or a second timer. - `POST /tx` queue admission: `try_send` on a full queue returns `429 OVERLOADED` with message `queue full`. -- Frame closure happens when direct inputs are drained, and also whenever batch closure happens. +- A logical-clock frame transition may drain zero or more directs. Batch closure separately creates the successor batch's structural first frame at the unchanged `safe_block`; block distance is the only reason logical frame time advances, not the only reason a frame row exists. - Batch closure is controlled by batch policy (size and/or deadline). - Preserve single-lane deterministic ordering. Do not introduce extra concurrency in hot-path ordering logic without explicit approval. @@ -235,10 +275,11 @@ Writer roles — one writer per table; reads over batch data go through the `val | Writer | Writes | |---|---| -| inclusion lane | `batches` (insert + `sealed_at_ms`), `frames`, `user_ops`, `sequenced_l2_txs`, `dumps`/`pending_snapshots` (batch close), `finalized_snapshot` (promotion) | -| input reader | `safe_inputs`, `l1_safe_head`, `safe_accepted_batches`, `deployment_identity`, `canonical_divergence` (poison marker, review R2) | -| recovery (startup) | `batches.invalidated_at_ms`, Tip reopen, scoped `pending_snapshots` clear, `wallet_nonce_watermark` (flush no-ops, write-before-broadcast) | -| batch submitter | `wallet_nonce_watermark` (write-before-broadcast, review R1a — its only write) | +| inclusion lane | `batches` (insert + `sealed_at_ms`), `frames`, `user_ops`, `sequenced_l2_txs`, `executed_inputs`, `dumps`/`pending_snapshots` (batch close), `finalized_snapshot` (promotion) | +| input reader | `safe_inputs`, `l1_safe_head`, `safe_accepted_batches`, `deployment_identity`, `canonical_divergence` (the divergence poison marker) | +| recovery (startup) | `batches.invalidated_at_ms`, Tip reopen, scoped `pending_snapshots` clear, derived `executed_inputs` suffix deletion, `wallet_nonce_watermark` (flush no-ops, write-before-broadcast) | +| history metadata (setup/recovery) | `history_state` — era/generation at baseline, generation bump in a non-empty standard-recovery cascade, rebuild application base + safe-input drain floor at initial finalized-snapshot registration | +| batch submitter | `wallet_nonce_watermark` (write-before-broadcast — its only write) | | egress (HTTP) | `dumps.lease_count` (leases) | | admin | `batch_policy` alpha knobs (`log_alpha`, `log_one_plus_alpha`) | | setup | `batch_policy.log_gas_price` + `log_gas_price_updated_at_ms` (first write; Fixed and Uniswap) | @@ -249,17 +290,28 @@ Writer roles — one writer per table; reads over batch data go through the `val - A frame's leading direct-input prefix is derivable from `sequenced_l2_txs` plus `frames.safe_block`. - Safe cursor/head values should be derived from persisted facts when possible, not duplicated as mutable fields. - Replay/catch-up uses persisted ordering plus persisted frame fee (`frames.fee`) to mirror inclusion semantics exactly. -- Cursor pagination for ordered L2 txs uses **SQLite rowid**, not count-based offsets. Holes from invalidated batches would break count-based pagination. +- Current cursor pagination for ordered L2 txs uses **SQLite rowid**. This is a + physical replay cursor and audit log, so invalidated rows remain and create + holes in the valid view. `executed_inputs` is the separate sparse, + current-canonical projection from physical rows to + `Application::executed_input_count()` offsets; invalidation deletes only the + doomed mappings so replacement history can reuse the same logical suffix. + The existing WS API has not switched coordinates yet. - Included user-op identity is tracked by application nonce logic; no DB uniqueness constraint (removed to allow resubmission after recovery). - **Reads over batch data go through `valid_batches`, `valid_closed_batches`, `valid_open_batch`, and `valid_sequenced_l2_txs` views.** These encapsulate the "exclude invalidated rows" filter so individual queries don't repeat it. Writers go to the base tables. - **`batches` row columns partition cleanly by writer.** `sealed_at_ms` is owned by the inclusion lane (set when closing a batch); `invalidated_at_ms` is owned by recovery (set during cascade). Each is write-once (NULL → non-NULL, never back) and enforced by triggers. The partial unique index `ux_single_valid_tip` guarantees at most one row has both NULL — the Tip. -- The inclusion lane is the **only writer** of open batch/frame state. `Storage::append_user_ops_chunk` and the `close_*` methods trust the in-memory `WriteHead`; the Tip-targeting triggers and the `pos_in_frame` PK catch stale-`WriteHead` bugs for **user ops**. **Direct-input sequencing has no structural uniqueness guard** (re-drain support requires duplicate `safe_input_index` across invalidated batches) — double-sequencing prevention rests on the lane's drain-cursor discipline and its startup re-derivation (see [`docs/invariants.md`](docs/invariants.md)). +- The inclusion lane is the **only writer** of open batch/frame state. SQLite is durable authority for those rows; `WriteHead` is a trusted coherent lane-local cache loaded from SQLite, advanced only after successful commits, and discarded on error/restart. It is reconstructible convenience, not an inter-component authority. `Storage::append_executed_user_ops_chunk` and the attributed `close_*` methods trust it without re-reading because the lane is the sole writer; the Tip-targeting triggers and the `pos_in_frame` PK catch stale-cache bugs for **user ops**. **Direct-input sequencing has no structural uniqueness guard** (re-drain support requires duplicate `safe_input_index` across invalidated batches) — double-sequencing prevention rests on the lane's drain-cursor discipline and its startup re-derivation. The sparse `executed_inputs` projection adds logical uniqueness and contiguity, but intentionally does not forbid duplicate physical safe-input rows across invalidated histories (see [`docs/invariants.md`](docs/invariants.md)). A more DB-derived, turn-stateless lane is a valid independent simplification to benchmark later, not part of the lane-reconciliation cutover. ## Type Boundaries - `SignedUserOp` — ingress/API signature domain (post-validation, pre-execution). - `ValidUserOp` — application execution domain (after validation boundary). - `SequencedL2Tx` — ordered replay/fanout domain (`UserOp | DirectInput`). +- `ExecutedInputCount` — canonical application-history boundary (`X` means the + next input is entry `X`), never a SQLite cursor. Checked arithmetic only. +- `ReplayL2TxRow` — crate-private named pairing of a physical DB cursor, + `SequencedL2Tx`, frame clock, and optional canonical attribution; do not + collapse these coordinates back into a positional tuple. - Keep DB-only helper types private to storage modules; prefer shared domain types at module boundaries. ## HTTP Endpoints @@ -279,7 +331,7 @@ Split by subcommand (the phase split). **`setup`** (required): - `CARTESI_SEQUENCER_BLOCKCHAIN_ID` - `CARTESI_SEQUENCER_APP_ADDRESS` - `CARTESI_SEQUENCER_BATCH_SUBMITTER_ADDRESS` (the submitter address — `setup` is L1-read-only and never signs). **Must be a dedicated address**: `setup`'s detection gate refuses if the submitter's wallet nonce is unsettled, so reusing a busy address (e.g. the contract deployer, whose deploy-tx tail isn't safe at setup time) false-positives. The devnet uses anvil account 9 (`DEVNET_SEQUENCER_ADDRESS`), distinct from the account-0 deployer. -- `CARTESI_SEQUENCER_CHECKPOINT_BLOCK` (optional, default `0` = genesis) — the trusted checkpoint machine's L1 inclusion block. `setup` refuses (typed `SetupRefuse`, exit 40 = run `setup --recovery`) if a previous instance left work past it. PR3 detects only; loading a non-genesis checkpoint machine is `setup --recovery` (PR5). +- `CARTESI_SEQUENCER_CHECKPOINT_BLOCK` (optional, default `0` = genesis) — the trusted checkpoint machine's L1 inclusion block. `setup` refuses (typed `SetupRefuse`, exit 40 = run `setup --recovery`) if a previous instance left work past it; plain `setup` detects only, and loading a non-genesis checkpoint machine is `setup --recovery`. **`run`** (required) — chain id / app address / submitter address are read from the DB `setup` pinned, not from args: @@ -287,7 +339,7 @@ Split by subcommand (the phase split). **`setup`** (required): - `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY` or `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE` **Optional** (names only — defaults and semantics are **owned by -[`sequencer/src/runtime/config.rs`](sequencer/src/runtime/config.rs)**; a +[`sequencer/src/commands/config.rs`](sequencer/src/commands/config.rs)**; a defaults list here drifted once already): `CARTESI_SEQUENCER_HTTP_ADDR`, `CARTESI_SEQUENCER_DATA_DIR`, `CARTESI_SEQUENCER_LONG_BLOCK_RANGE_ERROR_CODES`, `CARTESI_SEQUENCER_BATCH_SUBMITTER_IDLE_POLL_INTERVAL_MS`, `CARTESI_SEQUENCER_BATCH_SUBMITTER_CONFIRMATION_DEPTH`, `CARTESI_SEQUENCER_PREEMPTIVE_MARGIN_BLOCKS`, @@ -307,8 +359,30 @@ must be strictly below the danger threshold or startup refuses), - Keep application validation and execution deterministic for a given input/state. No `SystemTime::now()`, `HashMap` iteration order, or floating-point in consensus paths. - Surface user-facing errors via `ApiError` (in `http.rs`); keep internal failures descriptive but safe. - Avoid introducing heavy dependencies without strong reason. -- Documentation style: lean. Module headers (1–4 lines) + docs on public methods only when the contract isn't obvious from name+signature. Use inline comments for **why**, never for **what**. -- **Impossible states fail loud; they are never handled.** Cheap cross-module assertions of *real invariants* are encouraged (assert, trigger `RAISE`, typed error) — a loud crash is recoverable by design; silent divergence is not. Never add graceful fallbacks, neighbor re-validation, or silent absorbers (`INSERT OR IGNORE`, saturating decode of impossible data) for states the contracts rule out; and an assertion must check a real invariant, never an environmental assumption (clock monotonicity is the cautionary tale). Decision test and rationale: [`docs/invariants.md`](docs/invariants.md); trust boundaries: "Self-trust" in [`docs/threat-model/README.md`](docs/threat-model/README.md). +- Documentation style: lean. Module headers (1–4 lines) + docs on public methods only when the contract isn't obvious from name+signature. +- **Comment the non-obvious, not the self-evident.** Keep comments concise; avoid redundant and excessive inline commentary. Do not restate what the code already expresses. Explain the why, edge cases, invariants, and subtle behaviors that cannot be inferred from reading the code alone. +- Review-item codenames (finding/decision ids from past review ledgers) never appear in code comments or living docs — state the reason itself, or point at the invariant register entry that owns it. Invariant ids (`I1`–`I20`) are stable register references and are fine. +- **Impossible states fail loud; they are never handled.** Cheap cross-module assertions of *real invariants* are encouraged (assert, trigger `RAISE`, typed error). Failing loud is safety-preserving, not necessarily self-healing: transient failures may clear on restart, while a persistent invariant violation is terminal and may require inspection or cockroach recovery. Silent divergence is never acceptable. Never add graceful fallbacks, neighbor re-validation, or silent absorbers (`INSERT OR IGNORE`, saturating decode of impossible data) for states the contracts rule out; and an assertion must check a real invariant, never an environmental assumption (clock monotonicity is the cautionary tale). Decision test and rationale: [`docs/invariants.md`](docs/invariants.md); trust boundaries: "Self-trust" in [`docs/threat-model/README.md`](docs/threat-model/README.md). + +## Documentation Practice + +The corpus has two tenses, kept strictly apart: + +- **Living docs are timeless.** This file, `README.md`, `docs/protocol/`, + `docs/invariants.md`, `docs/recovery/`, `docs/snapshots/`, + `docs/threat-model/`, `docs/watchdog/`, and `docs/plans/` describe what is + true now and why — present tense, reasoning inline, no dates, no amendment + banners, no review codenames, no "previously/no longer". Each doc owns its + topic; others point at it rather than restating it. +- **History lives only in `docs/review/` and commit messages.** A review + ledger is append-only while its review is open. When it closes, distill it: + promote conclusions into the living docs, record settled decisions and + refuted proposals in [`docs/review/register.md`](docs/review/register.md), + and delete the process narration. Conclusions with reasoning outlive the + path taken to them. +- **Record deliberate absence once**, at the seam where someone would re-add + the mechanism, phrased as a positive design statement with its reason — + never as removal notices scattered across documents. ## Testing Guidance @@ -361,7 +435,7 @@ cargo run -p wallet-sequencer -- run - Add or update tests when logic changes. - Run at least `cargo check` before finishing. - Read `docs/recovery/` before touching recovery code, and `docs/threat-model/` before touching trust-boundary code. -- Check [`docs/invariants.md`](docs/invariants.md) before changing anything it lists as load-bearing, and the latest review ledger under [`docs/review/`](docs/review/) for known-open findings in the code you're about to touch. +- Check [`docs/invariants.md`](docs/invariants.md) before changing anything it lists as load-bearing, and [`docs/review/register.md`](docs/review/register.md) for open findings in the code you're about to touch and for decisions already settled or refuted. ### Ask First @@ -399,7 +473,8 @@ Before finishing a change, ensure: - [`CLAUDE.md`](CLAUDE.md) — shell setup, quick reference, pointer back here. - [`docs/protocol/`](docs/protocol/) — the authoritative protocol contracts: [`scheduler-semantics.md`](docs/protocol/scheduler-semantics.md) (the canonical acceptance algorithm, I1) and [`application-contract.md`](docs/protocol/application-contract.md) (the `Application` FFI trait contract). - [`docs/invariants.md`](docs/invariants.md) — register of cross-module invariants (what's load-bearing across files) + the fail-loud check policy. -- [`docs/review/`](docs/review/) — dated correctness-review ledgers; open findings, settled designs, work packages. +- [`docs/review/register.md`](docs/review/register.md) — the review register: open findings, settled decisions, and refuted proposals (do-not-re-propose), distilled from the dated ledgers beside it. +- [`docs/plans/`](docs/plans/) — the architecture decision record ([`2026-08-authority-boundary-adr.md`](docs/plans/2026-08-authority-boundary-adr.md)), active coordination tracks, and in-flight design handoffs. - [`docs/threat-model/README.md`](docs/threat-model/README.md) — trust boundaries, in-scope and out-of-scope threats. - [`docs/recovery/README.md`](docs/recovery/README.md) — recovery design, TLA+ formal verification, design history. - [`docs/snapshots/`](docs/snapshots/) — app snapshots: [`format.md`](docs/snapshots/format.md) (dump trait + wire format) and [`lifecycle.md`](docs/snapshots/lifecycle.md) (take/promote/GC/lease design + crash-safety). diff --git a/CLAUDE.md b/CLAUDE.md index a2056302..df9e35a0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -43,20 +43,25 @@ Rust edition 2024 / Axum API / SQLite (rusqlite, WAL) / EIP-712 signing / SSZ en `sequencer/src/` is organized by writer role; `storage/.rs` holds each role's storage half. -- `runtime/` — bootstrap, config, shutdown, shared clock. +- `commands/` — the operator command brackets (`run/` plus its worker + supervisor, `setup/`, `flush`) and their command-scoped `config` and + `error` taxonomy (incl. exit-code projection). +- `runtime/` — the runtime authority capabilities, consumed crate-wide: + the exclusive process lock and the runtime scope/shutdown machinery. - `ingress/` — public write path: `api.rs` (`POST /tx`) + `inclusion_lane/` (hot path). - `egress/` — internal read path: `api/` (WS subscribe + health) + `l2_tx_feed/`. - `l1/` — reader, submitter, fee oracle, provider, partition helper. - `recovery/` — startup preemptive-recovery procedure, runtime danger detector, mempool flusher. - `storage/` — SQLite persistence, split per writer role. -- `http.rs` — shared HTTP error type + `axum::serve` orchestration. +- `http.rs` — shared HTTP error type + `axum::serve` orchestration; `clock.rs` — the crate-wide wall clock. ## Before You Start Real Work - **[`AGENTS.md`](AGENTS.md)** — mission, requirements, invariants, duality, recovery, conventions, rules. - **[`docs/protocol/`](docs/protocol/)** — the authoritative protocol contracts: [`scheduler-semantics.md`](docs/protocol/scheduler-semantics.md) (canonical acceptance algorithm) and [`application-contract.md`](docs/protocol/application-contract.md) (the `Application` FFI trait). Read before touching the scheduler, the gold frontier, the fold, or an `Application` impl. - **[`docs/invariants.md`](docs/invariants.md)** — cross-module invariants register + the fail-loud check policy. Check it before changing anything it lists as load-bearing. -- **[`docs/review/`](docs/review/)** — dated correctness-review ledgers: known-open findings, settled designs, work packages. Check for open findings in code you're about to touch. +- **[`docs/review/register.md`](docs/review/register.md)** — the review register: open findings, settled decisions, refuted proposals (do-not-re-propose). Check it for open findings in code you're about to touch, and before proposing a mechanism or simplification. +- **[`docs/plans/`](docs/plans/)** — the [authority-boundary ADR](docs/plans/2026-08-authority-boundary-adr.md), active coordination tracks, and in-flight design handoffs. Check before starting work that might belong to a track. - **[`docs/threat-model/README.md`](docs/threat-model/README.md)** — trust boundaries and in-scope threats. - **[`docs/recovery/README.md`](docs/recovery/README.md)** — preemptive recovery design + TLA+ proofs. - **[`docs/snapshots/lifecycle.md`](docs/snapshots/lifecycle.md)** — snapshot lifecycle design + invariants (take/promote/GC, crash-safety). Read before touching the inclusion lane's safe-frontier/snapshot path. diff --git a/Cargo.lock b/Cargo.lock index 76220d61..942aacf4 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4268,6 +4268,7 @@ dependencies = [ "k256", "ruint", "serde", + "serde_json", "thiserror 2.0.19", ] diff --git a/README.md b/README.md index ec1306ec..7222d6d5 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,10 @@ A sequencer for Cartesi app-specific rollups. Provides low-latency soft confirma Rollup applications need fast transaction confirmations. Waiting for L1 finality on every user action (minutes) makes interactive applications impractical. The sequencer bridges this gap: it accepts signed user operations, immediately confirms them (soft confirmation), and asynchronously posts batches to L1. The application sees these batches posted on chain. -The core guarantee: **the off-chain sequencer and the rollup's on-chain scheduler produce identical execution order.** Users get instant feedback while the system converges to L1 truth. +The protocol objective is that, under supported honest operation, **the +off-chain sequencer predicts the same execution order the rollup's on-chain +scheduler later produces.** Soft confirmations are optimistic and may be +invalidated by the recovery cases below; L1 remains canonical truth. ## Two Chains Synchronizing @@ -22,7 +25,10 @@ Sequencer (off-chain) Scheduler (on-chain) user_ops=[D] execute D ``` -When things go well, the sequencer's chain and the scheduler's view converge. When they don't — batches arrive stale on L1 — the sequencer detects the divergence and recovers. +When things go well, the sequencer's chain and the scheduler's view converge. +When batches are becoming stale on L1, the sequencer detects the doomed suffix +and runs standard recovery. Terminal canonical divergence is the distinct +content-identity case below. ## Trust Model @@ -34,13 +40,28 @@ The sequencer is a **centralized, single-writer** system. It cannot steal funds **Direct inputs** (L1 → L2 messages, used for deposits) bypass the sequencer entirely. They are posted directly to L1 and are **uncensorable** by the sequencer — the scheduler drains them at every `safe_block` boundary. A censoring sequencer can delay when a direct input is executed (up to `MAX_WAIT_BLOCKS`, ~4h), but cannot prevent it. +During normal operation the sequencer advances logical frame time after five +newly-safe blocks have accumulated. That clock tick drains every covered direct +before later user ops and may also create an empty-direct frame to improve the +application-visible clock. Safe-head publication is best effort: if the node +exposes a multi-block jump, the sequencer creates one frame at the observed tip +and never fabricates intermediate frames. + Soft confirmations are an **optimistic prediction**: the sequencer also -cross-checks every batch the scheduler accepts on L1 against the batch it -sealed locally (a content-identity check), and refuses to operate further the -moment they differ. Detection happens when the divergent batch reaches L1 -*safe* finality, so soft confirmations issued inside that window (~2 L1 -epochs) can be built on already-diverged state — an inherent, bounded -property of the optimistic model. +cross-checks every at/above-anchor batch its off-chain scheduler simulation +accepts on L1 against the batch it sealed locally (a content-identity check). +When a foreign or byte-different landing reaches L1 *safe* finality and the +input reader ingests it, the same transaction records canonical divergence and +freezes the accepted frontier. The runtime stops when it next observes that +fact: the danger detector owns prompt process-wide reaction, while the +inclusion lane also refuses a poisoned projection if its existing time-gated +frontier read wins first. Every later boot refuses until an operator performs +cockroach recovery. User-op chunks committed before runtime observation may +still acknowledge and be rolled back. This check is a narrow +zombie/foreign-batch backstop, not proof that arbitrary application or +scheduler divergence cannot exist. It is not subsumed by the watchdog: the +marker freezes finalized-snapshot promotion, so the watchdog can legitimately +observe an unchanged finalized head and skip its state comparison. The third case is handled by the recovery subsystem. Batches that are too old when they reach L1 (`inclusion_block − safe_block ≥ MAX_WAIT_BLOCKS`) are skipped by the scheduler. This "staleness" poisons the nonce counter: all subsequent batches become unreachable regardless of their individual freshness. The sequencer detects this via a danger-zone threshold, preemptively goes offline, flushes the L1 mempool, and cascade-invalidates the doomed chain. See [`docs/recovery/`](docs/recovery/) for the full design, TLA+ formal verification, and design history. @@ -51,8 +72,8 @@ The sequencer trusts its own code is bug-free. Recovery means recovery from live The sequencer is designed to handle: - **L1 provider outages** — workers retry with exponential backoff. The inclusion lane and API continue operating locally. A wall-clock fallback detects when an outage pushes batches into the danger zone. -- **Process crashes** — recovery runs at startup. All recovery state is derived from SQLite (atomic transactions) and L1 safe state. No external coordination needed. -- **Extended downtime** — on restart, the sequencer syncs to the current L1 safe head, flushes if needed, and recovers. +- **Process crashes** — no operator action is needed: every boot derives any required recovery from SQLite and L1 safe state through the startup reducer, never assuming the previous exit was clean. A terminal death best-effort records its cause in the `terminal_faults` black box, which travels with the data directory for postmortems. +- **Extended downtime** — startup syncs to the current L1 safe head, flushes if needed, and recovers before admission; restart policy is the exit-code contract (a terminal exit means: do not restart, page an operator — the one manual remedy is a fresh-directory `setup --recovery` after canonical divergence). - **Adversarial L1 mempool** — block builders and private mempools are treated as adversarial. The recovery flusher consumes every pending nonce slot with a no-op so delayed "zombie" submissions cannot land later. ## Interfaces @@ -93,7 +114,9 @@ cargo run -p wallet-sequencer -- run ``` A third subcommand, **`flush-mempool`**, settles the batch-submitter wallet -nonce on demand (keyed operator tool). +nonce on demand (keyed operator tool). It is flush-only: it requires a +completed setup and no canonical divergence, and it never performs +Sync/Cascade or launches runtime workers. `setup` requires: `CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT`, `CARTESI_SEQUENCER_BLOCKCHAIN_ID`, `CARTESI_SEQUENCER_APP_ADDRESS`, `CARTESI_SEQUENCER_BATCH_SUBMITTER_ADDRESS`. `run` requires: `CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT`, `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY` (or `_FILE`); it refuses to boot until `setup` has completed. @@ -110,7 +133,7 @@ environment: CARTESI_SEQUENCER_ALLOW_INSECURE_RPC: "true" ``` -Process exit codes follow the R4 orchestrator contract: `0` clean shutdown, `10` restart (expect a recovery boot), `20` transient refusal (retry with backoff), `30` terminal (operator required — e.g. setup not complete, identity mismatch, canonical divergence), `1`/`101` unclassified/panic. +Process exit codes follow the orchestrator exit-code contract: `0` clean shutdown, `10` restart (expect a recovery boot), `20` transient refusal (retry with backoff), `30` terminal (operator required — e.g. setup not complete, identity mismatch, canonical divergence, persistent storage/application invariant failure), and `1` for an unclassified operational failure. Panics inside the command harness or supervised workers are projected to `30` under the fail-loud self-trust policy; `101` remains possible only before the harness can contain the command (for example, process/runtime initialization). Fixed protocol identity (EIP-712): @@ -209,7 +232,7 @@ released even on client disconnect. - `user_ops`: included user operations - `sequenced_l2_txs`: append-only ordered replay rows (`UserOp` xor `DirectInput`); inserting into `user_ops` also appends the corresponding replay row via trigger `trg_sequence_user_op` - `safe_inputs`: direct-input payload stream -- `batch_policy`: singleton knobs and constants for DA-style batch sizing and fee derivation; `batch_policy_derived` exposes `recommended_fee` and `batch_size_target`. Setup writes the first `log_gas_price` (and freshness stamp) for both Fixed and Uniswap modes; Uniswap then refreshes via the setup-pinned WETH/fee-token TWAP source. `log_slack = log(10)` applies the 10× safety margin in log space. Fixed local pricing has no oracle worker. Fees are app-token smallest units — initially USDC (6 decimals) for the wallet prototype — not a protocol-level USDC invariant. +- `batch_policy`: singleton knobs and constants for DA-style batch sizing and fee derivation; `batch_policy_derived` exposes `recommended_fee` and `batch_size_target`. Setup writes the first `log_gas_price` (and observation stamp) for both Fixed and Uniswap modes, failing if the initial Uniswap quote cannot be read. Fixed local pricing has no oracle worker; Uniswap starts from the persisted price and refreshes lazily via the setup-pinned WETH/fee-token TWAP source, retaining that price across transient source failures. `log_slack = log(10)` applies the 10× safety margin in log space. Fees are app-token smallest units — initially USDC (6 decimals) for the wallet prototype — not a protocol-level USDC invariant. ## Project Layout diff --git a/docs/invariants.md b/docs/invariants.md index 302a4908..d1fefffb 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -17,21 +17,82 @@ When you change anything listed under *enforced by*, re-check every line under - An invariant violation gets exactly one response: abort the operation loudly (assert, trigger `RAISE`, typed error). Cheap cross-module assertions at - boundaries are *encouraged*: a loud crash is recoverable by design - (orchestrator respawn + startup recovery), while a silently-tolerated bug - that externalizes (a signed batch, an ack, a feed event) is state divergence - — theft-equivalent and unrecoverable at runtime. + boundaries are *encouraged*. Failing loud is safety-preserving, not + necessarily self-healing: a transient failure may clear on restart, while a + persistent invalid row or state transition is terminal and can require + inspection or cockroach recovery. A silently-tolerated bug that externalizes + (a signed batch, an ack, a feed event) is state divergence — theft-equivalent + and unrecoverable at runtime. - **Never handle gracefully what cannot happen.** No fallback branches, no re-deriving a neighbor's answer to double-check it, no `Option`-handling for can't-be-`None`. One contract, one source of truth, no second code path. - **Never absorb silently.** No `INSERT OR IGNORE`, saturating decode, or `unwrap_or_default` on data the contracts make impossible; use the loud variant of the same operation. +- **Command admission is fact-derived; a contained terminal fault closes + in-process first.** Admission is governed by three facts, each with one + owner: the kernel process lock (concurrent owners; the exclusive OS-held + lock in `sequencer/src/runtime/process_lock.rs` makes + one-process-per-data-dir kernel-enforced, the controller retains it + through settlement, and nested work retains clones until it actually + stops), `setup_complete` (two-sided command ordering: setup/rebuild never + restart over a completed setup, run/flush never start before one), and + `canonical_divergence` (the one absorbing refusal — only a + fresh-directory cockroach rebuild proceeds). There is no lifecycle + admission state machine and no operator acknowledgement: standard + recovery is automatic — every run boots through the fact-derived + reducer — and restart policy after a terminal fault is the exit-code + contract (30 = do not restart, page), enforced by the supervisor; a + persistent fault re-detects fail-loud on any boot that reads it. The + only durable telemetry is the `terminal_faults` black box: append-only + terminal-cause rows, written best-effort and verdict-neutrally — + telemetry never changes a command's verdict, and nothing reads the + black box for decisions. Runtime admission re-runs the + reducer over one transactionally consistent fact set immediately before + the non-yielding launch block; the process lock plus the task-free + prepare phase make that read the decision's linearization. Baseline + schema+history creation and setup completion are each one + `synchronous=FULL` transaction. Runtime + containment remains classification-at-birth: detection CAS-elects one + reporter, sets the sticky containment bit, arms the independent + two-second abort watchdog, requests cooperative shutdown, and only then + appends the black box's terminal-cause row (best-effort telemetry — the + exit code and logs carry the verdict if it fails). The watchdog holds only a + weak process-lifetime witness; it aborts at the deadline exactly when a + controller, worker, or nested blocking operation still retains the + process lock. Cleanup polls all workers concurrently so one hung drain + cannot hide a terminal exit that must arm the bound. Ordinary + operator/recovery shutdown has no hard deadline. Externalization sites + (acks, L1 sends, WS frames) check the containment bit before emitting; + snapshot streams check it only at stream start today. A missed check is + bounded by the exit-code contract and by the I15 freeze triggers on the + tables they cover — partial structural backstops, not a barrier. +- **Maintenance is flush-only.** `flush-mempool` is an operator command, + not a run-reducer alias: it settles the wallet nonce and never acquires + Sync/Cascade semantics. It requires completed setup and no divergence. + (The old origin-restoration machinery is gone with the admission state + machine — there is no verdict state for a flush to erase, and a + successful wallet flush proves nothing about the rest of the runtime.) +- **Normal run repair and admission have one reducer boundary.** Local + absorbing facts are inspected before fallible provider facts. The pure run + decision performs at most one recovery phase, and every completed phase + returns to inspection. A successful flush contributes only an ephemeral + safe-block witness for the current boot attempt; Sync must catch the + persisted view up through that witness before Cascade, and a crash may safely + establish a fresh witness by flushing again. After the reducer decides to + admit, runtime preparation remains fallible and task-free; final + admission then re-runs the same reducer over one consistent fact set and + yields the single-use `RuntimeAdmission` witness consumed by the + infallible, non-yielding launch. Raw worker and HTTP launch surfaces are + crate-private; production app crates enter through `run`/`run_main`. No + refusal or retry can construct the capability. Mutation and output + authorization remains role-local at the durable boundaries documented + below; public low-level storage helpers are not an authority API. - An assertion must check a **real invariant** — true in every legitimate execution, including crash-recovery, replays, and clock steps — never an environmental assumption. (Cautionary tale: `sealed_at_ms >= created_at_ms` - was CHECK-enforced, wall-clock regression is legitimate, and the constraint - wedged recovery — review F8.) + was once CHECK-enforced; wall-clock regression is legitimate, and the + constraint wedged recovery before it was dropped.) Decision test for any proposed check: (a) real invariant? (b) near-zero cost? (c) fails loud with no alternative code path? Three yeses → write it. Any no → @@ -49,9 +110,10 @@ don't. accept/reject/ordering decisions for every input. (Known, documented exception: the predicate omits the two structural rejections — self-trust, since the simulator only runs over the sequencer's own well-formed batches; - the omission is documented in `scheduler-semantics.md`, not currently - test-pinned.) -- **Enforced by:** review + tests only. No mechanism. + the omission is documented in `scheduler-semantics.md` and test-pinned by + the I1 duality test in `sequencer-core/src/scheduler/mod.rs`, which asserts + the canonical fold and the predicate diverge exactly and only there.) +- **Enforced by:** review + the duality test. No structural mechanism. - **Depended on by:** everything — the gold frontier, recovery's cascade pivot, promotion, soft-confirmation honesty. - **Breaks:** silent permanent scheduler/sequencer divergence. @@ -61,12 +123,15 @@ don't. interleaves with storage-only side effects that can't move below the protocol layer — see the call-site comment). -### I2. Drain attribution: drained directs land in the new frame +### I2. Drain attribution: accumulated directs land in the clock-advanced frame -- **Holds:** at a safe-frontier advance, the newly-drained directs are - sequenced into the **new** frame, which is stamped with the **new** - `safe_block` (`close_frame_in`, `storage/ingress.rs`). Frame K's wire content - is therefore "directs ≤ S_K, then ops validated on top". +- **Holds:** when the observed safe head is at least five blocks beyond the + open frame clock, every newly-safe undrained direct is sequenced into the + **new** frame, which is stamped with the observed safe head + (`close_frame_in`, `storage/ingress.rs`). Directs may have accumulated across + several below-threshold observations. Frame K's wire content is therefore + "directs ≤ S_K, then ops validated on top"; a clock tick with no directs is + an empty-prefix instance of the same rule. - **Enforced by:** `close_frame_in` ordering; lane convention. - **Depended on by:** the duality (scheduler's drain-before-ops equals the flattened replay order); catch-up; the feed. @@ -75,10 +140,19 @@ don't. ### I3. Frame `safe_block`s are non-decreasing along the spine -- **Holds:** every frame opens at the current safe frontier, which only - advances. +- **Holds:** during an admitted live run, logical frame time advances directly + to the latest observed safe head `H` only when `H - S >= 5`, where `S` is the + open frame's persisted `safe_block`. An observation jump creates one frame at + `H` and resets the anchor; no intermediary frames are synthesized. Batch + closure may create a structural successor frame at the unchanged `S`, so + equality is valid. Bootstrap and recovery are anchoring transitions, not + live clock ticks: they may open a fresh Tip at a proven checkpoint/current + safe head without applying the five-block delta. - **Enforced by:** lane flow + `append_safe_inputs`' monotonicity asserts - (`storage/l1_inputs.rs`). + (`storage/l1_inputs.rs`) + + `ProtocolTiming::FRAME_CLOCK_INTERVAL_SAFE_BLOCKS` (homed with its timing + siblings in `sequencer-core/src/protocol.rs`; prose owner is the + scheduler-semantics frame-clock section). - **Depended on by:** `check_danger`'s arm ordering (see I4); the scheduler's within-batch monotonicity check; "if the frontier batch is fresh, all are". - **Breaks:** I4's guarantee evaporates; danger detection mis-orders. @@ -91,11 +165,10 @@ don't. - **Enforced by:** the arm order in `Storage::check_danger` (`storage/recovery.rs`) + I3. - **Depended on by:** the dispatch table's meaning (a `RecoverTip` boot may - skip the flush *because* nothing closed is doomed). **No longer load-bearing - for the pending clear**: since the F9 fix (2026-06-11) the clear is scoped - to `nonce >= pivot.nonce` in `cascade_and_reopen`, so a valid in-flight - closed batch's pending survives any cascade by construction, regardless of - arm order. + skip the flush *because* nothing closed is doomed). **Not load-bearing for + the pending clear**: the clear is scoped to `nonce >= pivot.nonce` in + `cascade_and_reopen`, so a valid in-flight closed batch's pending survives + any cascade by construction, regardless of arm order. - **Breaks:** a Tip-only cascade while a closed batch is doomed would leave the doomed batch un-cascaded until the next detector cycle (liveness lag, not the old crash-loop). @@ -117,7 +190,8 @@ don't. ### I6. A committed promotion implies an advanced drain - **Holds:** promotion is folded into the drain's transaction - (`close_frame_only_promoting`). + (`close_frame_only_promoting_with_executions`), together with canonical + direct-input attribution. - **Enforced by:** the single `write` tx in `storage/ingress.rs`; the standalone `Storage::promote_finalized` is test-only by policy. - **Depended on by:** crash-safety of the safe-frontier walk @@ -138,8 +212,11 @@ don't. - **Holds:** cold start registers the genesis dump as finalized and opens the genesis Tip; recovery reopens the Tip atomically across cascades. -- **Enforced by:** `Workers::spawn` order (`ensure_finalized_snapshot`, - `ensure_open_tip`) + recovery's in-tx reopen. +- **Enforced by:** `setup` atomically registers the genesis finalized snapshot + before its completion fact; the run reducer refuses a missing finalized + fact, opens a missing Tip only through its guarded `EnsureOpenTip` phase, + and recovery's cascade reopens in-transaction. `PreparedRuntime::prepare` + reasserts the snapshot artifact before admission. - **Depended on by:** catch-up's unconditional load path (`CatchUpError::NoSnapshot` is fail-loud, not a branch); the lane's `NoOpenTip` fail-loud load. @@ -147,10 +224,19 @@ don't. ### I9. Acceptance identity: "accepted nonce N" means "our valid batch N" -- **Holds:** by nonce **and content** since WP3 (review R2, 2026-06-12): every - fully-accepted landing is compared against the local valid closed batch at - that nonce — `keccak256(landed bytes)` vs the hash stamped at seal by the - same encode path the submitter broadcasts. +- **Holds:** by nonce **and content** — the **content-identity check**: every + landing at/above the batch-tree anchor that the off-chain + `scheduler_accepts` simulation accepts is compared against the local valid + closed batch at that nonce — `keccak256(landed bytes)` vs the hash stamped at + seal by the same encode path the submitter broadcasts. The exhaustive local + outcomes are `Match`, `Foreign` (no local valid closed batch), and `Mismatch` + (different bytes); the last two record divergence. +- **Why content, not identity, suffices:** batches deliberately carry no + identifier because content-equal copies are *effect-equal* — an accepted + batch's application effects depend on its inclusion block only through the + overdue force-drain, and for any fresh copy that force-executed prefix is + a subset of the first frame's drain, in the same queue order. Which + physical L1 transaction landed carries no semantic weight. - **Enforced by:** prevention — the flush resolving every wallet-nonce slot before a cascade reuses a nonce, anchored by the persisted watermark (I14); detection — the content-identity check in @@ -158,9 +244,23 @@ don't. `canonical_divergence` marker and freezes the frontier (I15). - **Depended on by:** the gold frontier, cascade pivot selection, promotion, local-state ↔ canonical-state agreement. -- **Breaks:** was silent divergence (review F1's zombie, F3's power-loss - re-seal); now a detected `CanonicalDivergence` refusal whose remedy is - cockroach recovery. +- **Breaks:** would be silent divergence (a zombie replay of our own stale tx + winning a nonce slot; a power-loss re-seal at the same nonce with different + content); instead it is a detected `CanonicalDivergence` refusal whose + remedy is cockroach recovery. +- **Completeness boundary:** the check completely enforces the accepted-batch + identity predicate above; it is intentionally not a general canonical/application + divergence oracle. It trusts collapsed history below the anchor and the + checkpoint application state, shares `scheduler_accepts` (including its + documented self-trust omissions), and does not independently detect bugs in + direct-input/user-op execution. A wrong-high cockroach checkpoint nonce is a + known example that can escape it. Absence of the marker therefore does not + prove global agreement. Detection is automatic once the landing is safe and + successfully ingested; repair is manual cockroach recovery, never standard + recovery. Detection latency is inherent to the optimistic model: the check + fires when the divergent landing reaches safe depth and is ingested, so + soft confirmations issued inside that window are built on already-diverged + state — bounded, and those confirmations are rollbackable by design. ### I10. Replay-offset sentinel: `0` means "from genesis" @@ -172,6 +272,10 @@ don't. - **Enforced by:** SQLite rowid semantics + append-only convention. - **Depended on by:** catch-up, the feed cursor, snapshot `l2_tx_index`. - **Breaks:** first transaction skipped or double-applied on replay. +- **Scope:** this is the current physical SQLite replay cursor. It is not the + canonical `Application::executed_input_count()` feed coordinate. The + canonical mapping is durable, but the public feed has not changed from rowid + pagination yet. ### I11. Own-batch safe inputs are sequenced but never executed or fanned out @@ -202,43 +306,82 @@ don't. - **Holds:** file create (fsync'd) before row insert; row delete before file delete; orphan *files* are acceptable and swept at startup. - **Enforced by:** ordering split between `storage/snapshot_dumps.rs` - (SQLite-only) and the lane's FS half (`inclusion_lane/snapshot.rs`) — the - module boundary *is* the ordering guarantee. + (SQLite-only) and the FS halves outside it (the lane's + `inclusion_lane/snapshot.rs`; the startup sweep in + `commands/run/startup_hygiene.rs`) — the module boundary *is* the ordering + guarantee. Startup and egress classify a missing or structurally corrupt + DB-referenced artifact as terminal; generic filesystem availability errors + remain operational. - **Depended on by:** `from_dump` at catch-up; the serving endpoints. -- **Breaks:** resume crash-loop. (Power-loss caveat until review R3 lands: - a non-fsynced row delete can rewind past a completed unlink — review F4.) +- **Breaks:** terminal startup refusal (or a terminal egress fault if detected + while serving), requiring inspection or cockroach recovery rather than an + automatic restart loop. ### I14. Watermark ≥ wallet nonce of every tx ever broadcast -- **Holds:** since WP2 (review R1a, 2026-06-11) — the watermark commits +- **Holds:** the **write-before-broadcast rule** — the watermark commits durably (`synchronous=FULL`) before any broadcast at a new nonce, - uniformly for batch txs and flush no-ops. + uniformly for batch txs and flush no-ops. A crash between commit and send + only over-covers (the flush later no-ops a never-used slot — harmless). - **Enforced by:** write-before-broadcast — `EthereumBatchPoster::submit_batches` raises through `WalletNonceWatermarkSink` before its first send; `MempoolFlusher::flush_and_wait` likewise before its no-ops, and refuses to complete until `safe >= watermark + 1`. - **Depended on by:** flush completeness, TLA+ Implementation Constraint 1, cascade soundness (I9). -- **Breaks:** zombie txs evade the flush — review F1. +- **Breaks:** zombie txs evade the flush — a dropped-locally but + network-surviving batch tx re-lands at a slot the recovery batch reuses, + and the scheduler executes invalidated content. ### I15. Divergence marker present ⇒ acceptance frontier frozen -- **Holds:** since WP3 (review R2, 2026-06-12). A fully-accepted landing that +- **Holds:** a fully-accepted landing that fails the content-identity check writes the `canonical_divergence` singleton **in the same transaction** as the sync that detected it, and `populate_safe_accepted_batches` returns early whenever the marker exists — so no acceptance row, no promotion, and no gold-frontier advance can ever happen past a detected divergence. -- **Enforced by:** the marker guard at the top of - `populate_safe_accepted_batches` + `check_danger`'s first arm - (`CanonicalDivergence`, ranked ahead of every other arm) + the - `Refuse(CanonicalDivergence)` startup dispatch. +- **Enforced by:** the `trg_*_frozen_on_divergence` trigger family + (`0001_schema.sql`) — specifically batch-tree writes, promotions, and + pending-snapshot clears RAISE in the engine while the marker exists. This is + the immediate persisted freeze for those named tables, not a general + user-op hot-path barrier. The typed error surface also includes the marker + guard at the top of + `populate_safe_accepted_batches` and `check_danger`'s first arm + (`CanonicalDivergence`, ranked ahead of every other arm). The run + reducer makes that ordering structural at boot: local inspection refuses + before any provider query, every completed phase re-enters inspection, and + each mutating phase transaction reasserts both its durable preconditions and + the absence of divergence before writing. The admission and preemptive TLA+ + models verify the controller ordering and slot/batch safety respectively. +- **Runtime reaction:** `check_danger` owns prompt process-wide reaction on its + two-second cadence. Independently, the inclusion lane's existing time-gated + SQLite read returns `SafeFrontierState::CanonicalDivergence` instead of an + `Open` frontier when the marker is already present. The lane then closes + intake, rejects queued work, and terminates before direct execution, + promotion, or the five-block rotation decision. This is opportunistic + refusal at an existing read, not another detector or a timing guarantee. + One bounded dequeue chunk is the fast-turn limit, so rejected traffic cannot + starve the read once its time gate is due. There is deliberately no + per-chunk marker query or extra poll. +- **Race bound:** a lane turn that already read `Open` may finish if the reader + commits divergence concurrently. Preventing that would require a lock or + transaction spanning application execution. Existing freeze triggers stop + conflicting batch-tree/promotion writes; the detector and next typed read + stop the process. A chunk committed before either runtime observation may + acknowledge and later roll back. +- **Watchdog boundary:** the freeze stops finalized promotion before the + offending landing becomes a comparable sequencer checkpoint. Because the + watchdog skips replay when the finalized inclusion block is unchanged, it + does not subsume this wire-identity detector. Conversely, the check does + not subsume the watchdog's broader independent application-state + comparison. - **Depended on by:** standard recovery never running on a diverged frontier (a flush+cascade there would compound the divergence); the lane never promoting a diverged landing; the remedy being cockroach recovery only. - **Breaks:** silent permanent scheduler/sequencer divergence — the - theft-equivalent failure the whole review centered on (F1/F3 residuals). -- **Anchor-aware frontier (PR5, 2026-06-26):** the content-identity check fires + theft-equivalent failure. +- **Anchor-aware frontier:** the content-identity check fires only at/above the batch-tree **anchor** ([I16](#i16-the-batch-tree-has-exactly-one-valid-parentless-root-carrying-the-deployments-anchor-nonce)). `populate_safe_accepted_batches` seeds its initial expected nonce from the anchor (0 for genesis — unchanged; `N'` for a cockroach-recovered deployment), @@ -253,7 +396,7 @@ don't. ### I16. The batch tree has exactly one valid parentless root, carrying the deployment's anchor nonce -- **Holds:** since PR5 (cockroach recovery, 2026-06-25). Every batch's nonce is +- **Holds:** every batch's nonce is `parent.nonce + 1`, except the single parentless root, which carries the `batch_tree_anchor` nonce — `0` for a genesis deployment, `N'` for a cockroach-recovered one (`setup --recovery` writes the anchor before the @@ -263,12 +406,11 @@ don't. invalidating the old root — so only one *valid* parentless root ever exists, invalidated ones coexisting. - **Enforced by:** `trg_enforce_nonce_contiguity` — its parentless arm is an - *exact* match `nonce == (SELECT nonce FROM batch_tree_anchor)` (tighter than - the pre-PR5 "must be 0"), plus an at-most-one-valid-parentless-root guard + *exact* match `nonce == (SELECT nonce FROM batch_tree_anchor)` (tighter + than a bare "must be 0"), plus an at-most-one-valid-parentless-root guard scoped to `invalidated_at_ms IS NULL`; `compute_next_nonce(None)` reads the same anchor; `trg_batch_tree_anchor_write_once` freezes the anchor once - `setup_complete` exists. Normal (anchor 0) deployments are byte-identical to - pre-PR5. + `setup_complete` exists. - **Depended on by:** the submitter resuming at the right nonce — `run` submits `valid_closed_batches` with `nonce >= frontier_nonce`, where `frontier_nonce` defaults to the anchor (`= N'`) while `safe_accepted_batches` is still empty @@ -280,3 +422,183 @@ don't. carries a nonce the scheduler rejects ⇒ the sequencer is wedged (never submits), or — worse, if defenses were absent — a recovered tree silently diverging from canonical L1 state. + +### I17. `WriteHead` is a coherent cache of the durable open Tip/frame + +- **Holds:** SQLite owns the durable open batch/frame facts. The single + inclusion lane loads one `WriteHead` from those facts at startup and threads + it through every open-state mutation. Storage validates fallible counter + advances before commit where needed, commits the durable rows, and mutates + the caller's cache only after transaction success; an error or restart + discards it and reloads from SQLite. +- **Enforced by:** the lane being the only open-state writer; + `load_current_write_head` being the single constructor for persisted state; + the `Storage::append_executed_user_ops_chunk`/attributed `close_*` update + ordering; and the Tip, + frame-position, FK, and PK constraints that fail loud on dangerous stale + cache writes. Direct-input uniqueness still depends on the lane's drain + cursor discipline because invalidated-history re-drain forbids a global + `safe_input_index` uniqueness constraint. +- **Depended on by:** the hot path avoiding a redundant SQLite re-read on every + chunk; batch-size/frame counters; safe-block drain attribution; every storage + method that trusts the passed head. +- **Breaks:** a stale cache can target the wrong Tip/frame, duplicate or skip a + position, or make live application order differ from durable replay order. + This is an internal bug and fails loud, never a runtime condition to repair. +- **Design latitude:** the cache is reconstructible convenience, not an + inter-component authority. Re-deriving more state from SQLite per turn may + simplify the lane, but is an independent benchmarked change rather than part + of the lane-reconciliation cutover. + +### I18. History metadata changes atomically with the history fact it describes + +- **Holds:** an authority-bearing initial setup/rebuild baseline creates the + schema, one immutable UUIDv4 `EraId`, and `RecoveryGeneration = 0` in one + `synchronous=FULL` transaction. Plain + setup starts with both bases zero; rebuild starts with + `base_executed_input_count = NULL` and `base_safe_input_index = NULL` + because neither the folded application nor its recovery-root cursor exists + yet. +- **Standard recovery:** `cascade_and_reopen` advances the generation exactly + once in its transaction iff it invalidates at least one valid batch. A + missing-Tip ensure or any other no-invalidation path leaves it unchanged. +- **Cockroach bind:** fill derives `K` from + `S'.executed_input_count()` and captures the recovery root's exclusive + safe-input cursor after sequencing its `<= C` padding. It binds both values + in the same transaction that registers the initial finalized snapshot. + `complete_setup` refuses while either base remains NULL or the finalized + snapshot is absent. The pair is write-once. On retry, a matching root Tip + plus that atomically bound snapshot/base pair is authoritative; it is not + re-compared with a later fold. +- **Durable drain floor:** the next-undrained cursor is the maximum of + `base_safe_input_index` and `MAX(valid safe_input_index) + 1`. Standard + recovery may invalidate the cockroach root and thereby remove its padding + from the valid view, but can never make inputs already represented by `S'` + drainable or executable again. NULL is interpreted as zero only while + setup has not completed (only a pre-completion rebuild fill can present a + NULL floor: plain setup binds base 0 in its baseline transaction, and + completion refuses while the base is NULL). +- **Coordinate separation:** `K` is an application-history boundary. It is + deliberately independent of snapshot `l2_tx_index` and the current rowid + feed cursor, which may include sequenced-but-not-executed cursor-padding + rows. The per-input projection is now durable, but the current public feed + still uses the physical cursor; its API/WS projection remains deferred. +- **Enforced by:** `baseline_migration` (`storage/open.rs`), the immutable-era, + write-once-base, and exact-`+1` schema triggers; `cascade_and_reopen` + (`storage/recovery.rs`); `insert_initial_finalized_dump` + (`storage/snapshot_dumps.rs`); and `complete_setup` + (`storage/lifecycle.rs`). +- **Depended on by:** standard-recovery discontinuity detection, honest + post-cockroach history availability, the canonical offset projection, and + the future Track 3 history-version/API protocol. +- **Breaks:** a client can mistake a rolled-back soft suffix for unchanged + history, or a rebuilt deployment can advertise an unavailable/incorrect + numeric prefix. Either silently diverges a mirror. +- **Operational boundary:** cockroach recovery remains an explicit + fresh/wiped-directory operator action. Retaining an early incomplete DB + reuses its still-unexposed era; a fail-loud partial-fill refusal requires a + wipe/retry and therefore a new unexposed era. No automated replacement, + clone detection, distributed fencing, or general resume state machine is + implied. + +### Do-not-simplify (deliberate shapes that look like cleanup targets) + +The refactorer-facing mirror of the register above — each of these *looks* +like a simplification and would break a registered invariant: + +- **Don't move filesystem work into `storage/snapshot_dumps.rs`** — the + module boundary *is* the GC crash-ordering guarantee (I13). +- **Don't reorder `check_danger`'s arms** or merge its two `find_*` helpers + into one that consults the Tip first — the closed-frontier-first order is + the dispatch table's meaning (I4). +- **Don't "deduplicate" promotion out of the drain transaction** — a + standalone promotion re-opens the promote-wedge crash loop (I6). +- **Don't filter own-batch rows out of `valid_sequenced_l2_txs`** — the + drain cursor is `MAX(safe_input_index)+1` over those very rows; a view + filter would rewind it and re-drain. Sender filtering stays at the + consumers (I11). +- **Don't replace the rowid offset with count-based pagination** — + invalidated-batch holes and the 0-sentinel depend on current physical + behavior (I10). +- **Don't move snapshot GC off the promotion path** to an idle loop or a + dedicated worker — promotion-coupled GC is starvation-proof and + single-writer by design (`docs/snapshots/lifecycle.md`). +- **Don't add internal retry loops to the flusher/submitter for provider + errors** — the orchestrator respawn is the retry mechanism; internal + retries mask exactly the failures the danger machinery routes on. +- **Don't unify the two staleness references** (inclusion-relative vs + current-relative) — deliberately different formulas for different + questions. + +### I19. Application progress advances only at the shared execution boundary + +- **Holds:** `ApplicationProgress` is the pair + `(ExecutedInputCount, last_executed_safe_block)`. Count zero implies clock + zero. A successful canonical application input returns its pre-execution + count as the offset and commits exactly `(count + 1, max(clock, + input_clock))`; rejection changes neither field. `AppError` is fatal and + defines no canonical successor. +- **Enforced by:** raw `apply_*` hooks and mutable progress access require + distinct borrowed opaque capabilities constructible only by the shared + execution functions. The boundary preflights count overflow, checks progress + unchanged after validation and after a hook on both `Ok` and `Err`, and + re-reads the immutable getter after commit to assert accessor coherence. +- **Depended on by:** the canonical scheduler, inclusion lane, catch-up, + recovery fold, cockroach base `K`, durable execution attribution, and the + future Track 3 API projection. +- **Breaks:** an input can be applied without advancing history, an offset can + advance twice, or recovery can derive the wrong checkpoint clock — silent + application-history divergence. +- **Scope:** application-specific mutation and determinism remain self-trusted. + A failing hook is not rolled back; every production caller terminates that + path and discards the instance. + +### I20. Canonical execution offsets are an atomic projection of valid history + +- **Holds:** `sequenced_l2_txs` remains the append-only physical replay/audit + log. `executed_inputs` is a separate sparse projection for the current valid + history: every user op and non-batch-submitter direct input that executes has + exactly one mapping from its physical row to the pre-execution + `ExecutedInputCount`; batch envelopes and cockroach-root cursor-padding rows + have none. Current mappings occupy the contiguous logical interval `[K, H)`. +- **Creation atomicity:** a user-op chunk inserts its `user_ops`, trigger-created + physical rows, and explicit execution mappings in the same FULL transaction + that authorizes acknowledgements. A slow reconciliation turn inserts its + direct physical rows, mappings, frame rotation, and any snapshot promotion + in one transaction. The lane carries offsets attached to executed values, so + an included input cannot be persisted without its receipt. +- **Recovery semantics:** suffix invalidation retains physical audit rows but + deletes their derived mappings in the same transaction that advances + `RecoveryGeneration` and opens the replacement Tip. This rewinds `H` + naturally; replacement inputs reuse the suffix offsets under the new + generation. The global logical UNIQUE constraint and next-offset trigger + make a duplicate, gap, or out-of-order creation fail loud. Cockroach padding + stays outside the projection, and the durable safe-input floor prevents it + from being attributed later. +- **Snapshot/replay agreement:** every pending/finalized snapshot row stores + both physical `l2_tx_index` and canonical `executed_input_count`. Snapshot + registration asserts its count equals storage-derived `H`; startup compares + the loaded application's count with the row; catch-up then checks each + physical row's expected mapping before executing it. Missing, extra, or + wrong mappings are terminal invariant failures, never repaired/backfilled. +- **Enforced by:** `ExecutedInputCount` receipts + (`sequencer-core/src/application/mod.rs`); attributed lane/storage APIs + (`ingress/inclusion_lane/`, `storage/ingress.rs`, + `storage/mutations.rs`); `executed_inputs` constraints and invalidation + trigger (`storage/migrations/0001_schema.sql`); storage-derived `H` + (`storage/history.rs`); snapshot count checks + (`storage/snapshot_dumps.rs`); and pre-execution catch-up checks + (`ingress/inclusion_lane/catch_up.rs`). +- **Depended on by:** restart determinism, standard-recovery rollback/reuse, + post-cockroach continuation at `K`, snapshot coherence, and the future + canonical-offset HTTP/WS protocol. +- **Breaks:** the same numeric offset can name the wrong application input, or + a restart can apply a different prefix than live execution—silent mirror or + canonical-state divergence. +- **Performance boundary:** deriving `H` is a covering lookup over the logical + UNIQUE index. Recovery deletes only its doomed projection suffix; it does + not scan invalid physical history on every hot-path insertion. Direct + execution receipt accumulation and classification live in the already-slow + L1 reconciliation regime; the user-op hot path adds one chunk-level mapping + query and inserts inside its existing durability transaction, not another + fsync or actor. diff --git a/docs/plans/2026-07-coordination-tracks.md b/docs/plans/2026-07-coordination-tracks.md new file mode 100644 index 00000000..967e039e --- /dev/null +++ b/docs/plans/2026-07-coordination-tracks.md @@ -0,0 +1,112 @@ +# Coordination Tracks + +**Status:** active plan of record. Tick / annotate as work lands; when a +track completes, move its durable outcomes into the normative docs and +collapse its entry here. + +Context: Bart is building **libdex**, a native (non-CM) app whose backing +storage is an mmap'd flat buffer, and will reimplement the scheduler in C++. +Two of his needs shape this plan: bit-exact dapp-state mirroring from the WS +feed, and an ergonomic/efficient `Application` dump story. We break APIs +freely at this stage — no backward-compatibility constraints. + +| # | Track | Owner | Status | +|---|-------|-------|--------| +| 1 | WS context fields + L1 provenance (PR #26) | Stephen | **done** — merged to main | +| 2 | Restore `docs/review/` ledger + this plan | us | **done** | +| 3 | Feed & replay protocol redesign | us (design) → us/Stephen (impl) | **storage foundation landed; public API open** — the [Track 3 ordered handoff](2026-07-track3-feed-replay-design.md#7-ordered-implementation-handoff) exclusively owns its sequence and decision gates | +| 4 | Storage decode policy | us | **done** — fail-loud for contract-impossible values; the named `saturating_query_bound` only where clamping preserves the predicate (policy lives in `storage/convert.rs` + the invariants check policy) | +| 5 | Fee exponentiation LUT | us | **deferred** — decided exact-floor if built (the table *is* the spec, algorithm-free; replay continuity across the upgrade explicitly not preserved); a separate pending design decision may make log-space fees defunct — revisit after syncing with Bart | +| 6 | Dump / `Application` API redesign | us + Bart | **design drafted, under review with Bart** — [`2026-07-track6-dump-api-design.md`](2026-07-track6-dump-api-design.md); see the constraints below | +| 7 | LLM context-engineering review | us | **done** — skills/agents/settings homed in-tree; the docs-practice rules live in AGENTS.md | +| 8 | Terminal-containment structural consolidation | us | **done** — superseded by and landed through the [authority-boundary ADR](2026-08-authority-boundary-adr.md) | + +**Current campaign order:** + +1. Land the authority-boundary + durable-history-foundation branch (squashed, + review complete — ready for its PR against main). +2. Implement Track 3's public protocol on a focused successor branch. +3. Track 6 implementation after the design settles with Bart. +4. Track 5 (fee LUT) only after the log-space-fees decision. + +Deferred (revisit with libdex rollout): multi-file/tar snapshot serving +(`docs/snapshots/lifecycle.md` known limitation), pending-snapshot-pool cap. + +## Track 3 — Feed & replay protocol redesign + +The current protocol grew ad hoc; the redesign is type-first and covers the +whole consumer data-access story: paginated finalized-history endpoints plus +the live subscription, composable without races. The +[design doc](2026-07-track3-feed-replay-design.md) owns the requirements and +the ordered implementation handoff; the storage/recovery foundation +(era/generation metadata, canonical `ExecutedInputCount` attribution, +snapshot/catch-up verification) is landed, while `GET /history-version`, +replay routes, gold-boundary projection, and WS v2 remain open. + +Settled decisions the implementation must respect: + +- **Feed coordinate:** `Application::executed_input_count()`, not SQLite + rowid. An application at count `X` subscribes at `X`, consumes entry `X`, + advances to `X + 1`. Standard recovery may reuse suffix offsets under a new + generation; cockroach recovery records the folded count `K` as the era's + available-history base, and requests below `K` fail with `available_from` + plus the bootstrap recipe. +- **Discontinuity detection is pull-based.** A crash or danger-detector exit + cannot send a farewell frame, so the load-bearing contract is the required + subscription claim `{era_id, recovery_generation, offset}` plus a + current-pair endpoint; in-band disconnect errors are best-effort only. + Bart confirmed the scalar generation contract (2026-07-28); the `EraId` + generalization and changed-era bootstrap behavior still need his consumer + review and are not attributed to that confirmation. +- **Event framing:** per-row denormalized context (as shipped in PR #26); + no `FrameSealed`/`BatchSealed` boundary events unless a consumer + demonstrates the row context cannot express its need. +- **Clock:** application time is safe-block based. Direct inputs execute at + their exact inclusion block; user ops at their frame's safe block. + `block_timestamp` may ride as provenance but is never an application + transition input (see the application contract). + +## Track 5 — Fee exponentiation LUT (deferred) + +`fee_to_linear` is consensus-critical (scheduler fold, guest agreement, app +fee charging) and must be bit-identical across the Rust sequencer, the +RISC-V guest, and Bart's C++ scheduler. Today's implementation shares a +15-entry squares table but also requires reproducing `fixed_mul` exactly — +256×256→512 widening multiply, `>> 64`, truncate, LSB-first accumulation +with floor after each multiply — which is unreasonable to demand of a port. +If built, the shape is decided: a full lookup table of exact +`floor((129/128)^n)` bignum values for every legal exponent (~17k entries ≈ +550 KB). The checked-in table *is* the cross-implementation spec artifact +(algorithm-free, golden-hash-tested); `build.rs` verifies rather than +generates; C++ consumes the same file byte-identically; and replay +continuity across the upgrade is explicitly not preserved. Do not implement +until the pending log-space-fees decision lands (with Bart). + +## Track 6 — Dump / `Application` API redesign + +The `create_dump` / `from_dump` / `delete_dump` / `state_file_in_dump` +surface is a leaky projection of what the inclusion lane needs; the design +doc inventories the real requirements (atomic crash-durable checkpoint, +startup reconstruct, disposal, serve-canonical-bytes-without-instantiating). +Constraints established for the design review with Bart: + +- The CM emulator has **no commit/revert** — its API is + `load / store / clone_stored / remove_stored`; commit/revert stay + sequencer-side (DB row as commit point; older-dump+replay as revert). The + genuinely missing primitive is **cheap clone** (reflink with graceful + full-copy fallback; hardlink suitability for a mutable working image is + disputed — settle in the design review). +- **Durability postures are opposite and must not be silently inherited:** + the sequencer mandates fsync inside the checkpoint (I13); CM/Dave fsync + nothing and compensate with hash-on-load. Keep the app-fsync posture; a + CoW implementation must fsync what reflink leaves unsynced. (CM PR #398's + durable `rename_stored`/`remove_stored` narrows this gap — re-check before + finalizing the crash-safety section.) +- **libdex layout constraint to communicate early:** the served canonical + file must byte-match the canonical machine's `inspect_state` output (the + watchdog byte-compares). A raw mmap buffer with allocator padding or + pointer-valued fields breaks that: either the buffer layout is itself + canonical, or libdex needs a separate canonical projection. + +Deliverable: design settled jointly with the Track 3 doc, in front of Bart +together — his on-disk layout decision depends on both. diff --git a/docs/plans/2026-07-track3-feed-replay-design.md b/docs/plans/2026-07-track3-feed-replay-design.md new file mode 100644 index 00000000..78b71470 --- /dev/null +++ b/docs/plans/2026-07-track3-feed-replay-design.md @@ -0,0 +1,372 @@ +# Feed & Replay Protocol Design (Track 3) + +**Status: accepted architecture; storage/execution foundation landed; public +protocol open.** The team accepted the architecture recorded here; Bart's +consumer review remains open only for the decision gates named in §8. The +history identity `(EraId, RecoveryGeneration)` was accepted 2026-08-01; the +authoritative `Application::executed_input_count` offset and cockroach-base +semantics were accepted 2026-08-02. Durable DB metadata, the typed +application-execution boundary, per-input canonical attribution, and +snapshot/catch-up agreement are +landed. `GET /history-version`, replay routes, and the WS projection remain +open. The consumer bootstrap and `/inputs` questions remain decision gates. +The completed protocol will supersede the ad-hoc WS protocol and close the +open WS invalidation-contract finding (see the review register) only +when the ordered handoff in §7 is complete. At that point, the +normative parts graduate into `docs/protocol/` and the README is rewritten. + +## 1. Motivation + +The current protocol is an unframed infinite stream of a two-variant enum over +a WS socket, with every session-level signal smuggled into transport close +frames. Three defects drive the redesign: + +- **The open invalidation-contract finding (register):** no + invalidation/rollback signal. Today, recovery + cascades hide already-streamed rowid-addressed rows and re-sequence + replacements at higher rowids under a reused batch nonce. A cursor-resumed + mirror can silently diverge. +- **No historical bootstrap:** the catch-up window is shallow; a consumer + cannot build state from genesis over the feed. +- **No control plane:** catch-up-to-live is unmarked and continuity errors are + prose close reasons rather than typed messages. + +Consumer model (Bart's libdex is the concrete instance): replay finalized +history via HTTP, then subscribe for the soft tip while maintaining a bit-exact +mirror of application state. Recovery and rebuild boundaries must be detectable +and have explicit remediation. + +## 2. Concepts and coordinates + +- **Input-box coordinate `input_index: u64`** — position in `safe_inputs` + (per-application InputBox order). Append-only and sourced from L1 safe blocks. +- **Feed coordinate `offset: ExecutedInputCount`** — the authoritative + `Application::executed_input_count()` boundary, starting at zero. An + application at `X` is ready to consume history entry `X`; applying that + entry advances it to `X + 1`. SQLite now stores this as a sparse canonical + attribution beside its append-only physical rowid replay log. The current + public feed still exposes rowid and changes only at the later API cutover. +- **Era base `K`** — the smallest feed offset locally available in this era. + Genesis setup starts at zero. Cockroach recovery sets it to + `S'.executed_input_count()` after the fold; absolute offsets continue, but + the unavailable prefix is not reconstructed. `K` is application history, + not the snapshot's physical `l2_tx_index`: recovery cursor-padding rows may + advance the latter without executing an application input. +- **Gold boundary `G`** — within one era, the exclusive executed-input count + after the scheduler-accepted prefix. Entries with `offset < G` cannot be + invalidated; `G` only advances. +- **Era ID `e`** — random durable UUIDv4 minted write-once in one era's + baseline transaction. Cockroach recovery/fresh setup creates a new era + because the rebuilt DB cannot serve the prior era's ordered L2 history from + genesis. +- **Recovery generation `g: u64`** — soft-suffix reality version within one + era. Bumped exactly once by a standard-recovery transaction iff it + invalidates at least one valid batch. Entries with `offset < G` are + generation-free within that era. +- **History version `(e, g)`** — equality/discontinuity token carried by the + protocol. It is not a globally ordered number. +- **Live head `H`** — the current application's exclusive + `executed_input_count`; locally available entries occupy `[K, H)`. + +## 3. History-version semantics + +- `EraId` is a 16-byte UUIDv4 newtype, persisted write-once per era and exposed + as canonical lowercase hyphenated JSON. Store `created_at` separately; a + bare timestamp is not collision-resistant under clock rollback or + simultaneous setup. +- `RecoveryGeneration` starts at zero and increments exactly once in the same + transaction iff standard recovery invalidates at least one valid batch. + Ensuring/reopening a missing Tip without invalidation does not bump it. +- Clean restart and inspection that admits without changing history change + neither field. +- Cockroach recovery/fresh setup mints a new era and resets generation to + zero. An interrupted attempt that retains its incomplete DB reuses the + already-minted, externally unexposed era. A fail-loud partial-fill refusal + requires an operator wipe/retry and therefore mints another unexposed era. + A new era is an explicit operator-driven setup/rebuild action; there is no + in-place rotation tool, automated DB replacement, implicit clone detection, + distributed fencing, or partial-fill resume protocol. +- Copying an initialized DB copies its era too. Arbitrary clone-and-run is + unsupported. Operating copied state as a new era requires explicit + fresh/wiped-directory setup/rebuild; detecting uncoordinated clones requires + external authority. +- The protocol will expose the pair through `GET /history-version`; every + subscribe request will claim it. WS responses repeat only the recovery + generation, including best-effort in-band `discontinuity` events. None of + those API changes is part of the landed storage foundation. + +Within the same era, a stale generation means “discard and replay the soft +suffix.” Standard recovery restores the retained application state, so its +count rolls back, then advances it over replacement force-drained directs; the +same suffix offsets may therefore name different inputs under the new +generation. A changed era means the client reacquires a current-era +snapshot/bootstrap even if its old state's numeric count happens to be in the +new era's range. + +Cockroach recovery now leaves the rebuild base NULL at baseline creation, then +binds `K = S'.executed_input_count()` in the same transaction that registers +the initial finalized snapshot. Setup completion refuses until both exist. +Requests below `K` receive a typed `history_unavailable` response carrying +`available_from = K` and the bootstrap recipe. This preserves the absolute +application coordinate without claiming that the rebuilt DB can serve the +lost prefix. + +### 3.1 Landed storage representation + +The physical and logical coordinates deliberately remain separate: + +- `sequenced_l2_txs.offset` is the append-only SQLite replay/audit cursor. + Invalidated rows remain, and batch-envelope/cockroach-padding rows exist even + though the application does not execute them. +- `executed_inputs` is a sparse **current-canonical projection** from an + executable physical row to its pre-execution `ExecutedInputCount`. User-op + and direct mappings commit atomically with their existing durability + transaction; envelopes and padding have no row. +- Standard recovery retains physical audit history but deletes the invalidated + mapping suffix in the same transaction that bumps generation and opens the + replacement Tip. `H` therefore rolls back without scanning invalid physical + history, and replacements reuse the same suffix offsets under the new + generation. +- Snapshot rows store both physical `l2_tx_index` and canonical + `executed_input_count`. Registration checks the app count against + storage-derived `H`; startup checks the loaded dump against the snapshot row; + catch-up checks every mapping before executing its physical row. + +There is no backfill, repair, or neighbor-derived fallback. A missing, extra, +or wrong attribution is a terminal self-invariant failure. This keeps the +public API cutover a projection over already-correct durable values rather than +the moment those values first become authoritative. + +## 4. Historical replay endpoints (HTTP, paginated, finalized-only) + +Both endpoints serve immutable entries and therefore carry no recovery +generation. Every response carries `era_id`; an entry already returned within +that era never changes. Tail pages and their current end/gold metadata can +advance, so clients and caches must revalidate the current tail rather than +treating every page response as immutable. A client must never splice pages +from different eras. + +### 4.1 `GET /inputs?from_index=N&limit=K` + +Raw InputBox order over `safe_inputs`: direct inputs and our own batches, +exactly as L1 ordered them, with PR #26 provenance: + +```json +{ + "era_id": "550e8400-e29b-41d4-a716-446655440000", + "items": [ + { + "input_index": 7, + "sender": "0x...", + "payload": "0x...", + "block_number": 123, + "block_timestamp": 1700000000, + "transaction_hash": "0x..." + } + ], + "next_index": 8, + "end_index": 41 +} +``` + +`end_index` is the current exclusive upper bound. Rows appear once the L1 safe +head passes them. + +### 4.2 `GET /l2-txs?from_offset=N&limit=K` + +Feed order capped at the gold boundary. Items use the same shapes as WS data +events so replay pages and live events are interchangeable mirror inputs: + +```json +{ + "era_id": "550e8400-e29b-41d4-a716-446655440000", + "items": [], + "next_offset": 101, + "gold_boundary": 250 +} +``` + +`from_offset = N` is inclusive: the first item, if present, is history entry +`N`; `next_offset` is the application count after all returned entries. The +server never serves `offset >= G` here. A dedicated indexed query computes `G` +per page. Because it only advances, a stale read is conservative. Requests +below the era base fail with `history_unavailable` rather than pretending the +missing prefix is empty. + +## 5. WS subscription v2 + +### 5.1 Handshake and continuity + +`GET /ws/subscribe?from_offset=N&era_id=e&recovery_generation=g` upgrades. All +three coordinates are required: the client claims both the history reality it +holds and the exact application boundary it is ready to execute. + +```json +{ + "kind": "hello", + "recovery_generation": 4, + "available_from": 100, + "live_head": 312, + "gold_boundary": 250, + "max_subscribers": 64 +} +``` + +- Every server WS message carries the current `recovery_generation`; no WS + response repeats `EraId`. The era is an admission credential, and a client + obtains the current value through the bootstrap/history-version path. +- If `era_id` differs, send `era_changed`, then close 1008. Old application + state cannot be resumed merely because its count is numerically in range; + the client reacquires the current-era snapshot/bootstrap. +- If the era matches but `recovery_generation` differs, send + `stale_generation` with the current generation, then close 1008. The client + rebuilds only its soft suffix above its last known `G`. +- If `from_offset < K`, send `history_unavailable { available_from: K, ... }`, + then close 1008. The client acquires the snapshot/bootstrap at `K`. + +### 5.2 Depth guarantee + +Within one era, every `from_offset` in `[G, H]` is serveable: WS supplies the +soft-history entries in `[G, H)`, while a request exactly at `H` joins live and +waits for the next input. Section 8 owns the policy for offsets above `H`. HTTP +replay covers entries below `G`. A request in `[K, G)` receives: + +```json +{ + "kind": "error", + "recovery_generation": 4, + "error": "below_gold_boundary", + "gold_boundary": 250 +} +``` + +then a 1008 close. The client loop is: + +1. Page `/l2-txs` until history is exhausted at `G0`, verifying one `era_id` + across every page. +2. Subscribe from `G0` with the pair from `/history-version` (or the + current-era bootstrap response). +3. On `stale_generation`, discard the soft suffix and return to step 1. On + `below_gold_boundary`, return to step 1. On `era_changed`, reacquire the + current-era snapshot/bootstrap and restart. + +There is no same-era continuity hole if `G` advances between replay and +subscribe: entries in `[G0, G)` remain finalized and serveable over HTTP, and +the typed `below_gold_boundary` response sends the client back through that +replay. Both HTTP and WS use the same boundary convention: an application at +count `X` requests `X`, and the first returned input is entry `X`. + +### 5.3 Events + +Data events retain PR #26's denormalized row context (`user_op` / +`direct_input` with nonce, safe block, batch nonce, input index, block +timestamp, and transaction hash). Bart confirmed this field set suffices for a +bit-exact mirror. Normalized frame/batch boundary events remain rejected until +a consumer demonstrates a need. + +Every event carries `recovery_generation`. Control events join the same tagged +enum: + +- `hello` — §5.1; +- `live` — sent once catch-up reaches the live head; +- `discontinuity { recovery_generation }` — best-effort only; reconnect + validation is load-bearing; +- `error { error, ... }` — typed policy error followed by close 1008. + +### 5.4 Wire format + +JSON text frames, serde-tagged by `kind`, with mandatory new fields. We break +old clients freely. `WsTxMessage = BroadcastTxMessage` remains the shared +exhaustive SDK/server type. + +## 6. What this replaces + +- `WS_CATCHUP_WINDOW_EXCEEDED_REASON` and the `live_start_offset` prose close + reason — replaced by `below_gold_boundary` plus `/l2-txs`. +- The interim “rebuild on any socket drop” rule — replaced by mandatory + history-version validation on every subscription. +- README's WS section — rewritten as part of the WS v2 cutover. + +## 7. Ordered implementation handoff + +> **At-risk dependency:** Bart's 2026-07-28 +> confirmation covers the scalar generation contract only. The `EraId` leg — +> changed-era rejection and current-era bootstrap behavior — is explicitly +> unconfirmed by the consumer. The durable schema slice is cheap to carry, +> but treat the era semantics as provisional in every wire-projection step +> below: the wire form is the part that cannot be cheaply migrated once a +> consumer depends on it, so it must not ship ahead of Bart's review. + +Track 3 owns the remaining consumer-facing history/feed protocol. The +storage/execution foundation in §3.1 is complete. The current public rowid +contract remains unchanged until the API cutover lands as one deployable +protocol boundary. + +1. **Resolve the consumer decision gates.** With Bart, settle the `/inputs` + representation and the current-era post-cockroach bootstrap artifact/API. + Decide future-offset behavior before WS v2 and replay authentication/rate + limits before public exposure. Boundary events remain excluded unless a + consumer demonstrates a need. Track 3 owns the consumer-visible bootstrap + contract; Track 6 owns the dump/image representation it may serve. +2. **Add the typed history read/API foundation.** Define shared history-claim, + page, and policy-error types. Logical boundaries use `EraId`, + `RecoveryGeneration`, and `ExecutedInputCount`, not interchangeable raw + `u64` cursors. Back them with one internally consistent SQLite read of + `(e, g, K, H, G)`, canonical inclusive pagination through + `executed_inputs`, and raw safe-input pagination with provenance. +3. **Implement HTTP bootstrap and finalized replay.** Add + `GET /history-version`, `/inputs`, and `/l2-txs`, including era-tagged pages, + the gold-boundary query, typed below-`K` bootstrap errors, and + pagination/provenance/forced-cascade immutability tests. +4. **Cut `/ws/subscribe` over to v2.** Require + `(EraId, RecoveryGeneration, ExecutedInputCount)`, validate the claim before + admission, serve canonical-coordinate soft history for every admitted offset + at or above `G`, emit the typed hello/control/error vocabulary, carry + `RecoveryGeneration` on every response, and remove the + physical-rowid/string-close contract. Update the SDK and harness in the same + cutover. +5. **Prove and close the feature.** Replace the behavior-pinning E2Es with + stale-generation, era-change, below-base, logical-suffix-reuse, + pagination-hole, and replay-to-live race cases; rewrite the README; graduate + the normative protocol text; remeasure submit-to-matching-WS-event latency; + then mark the WS invalidation-contract finding fixed in the register. + +Steps 2 and the internal part of 3 can begin before the consumer questions +close. Do not freeze the public below-`K` recovery recipe or `/inputs` response +shape until the relevant questions do. Feed output must come from committed +valid SQLite history and match the requested `HistoryVersion`; it does not +depend on a global runtime actor. + +## 8. Decision gates and revisit triggers + +1. **Boundary events:** revisit if a consumer needs frame/batch boundaries the + denormalized rows cannot express. +2. **`/inputs` shape:** confirm how Bart reconciles raw L1 order against feed + order and settlement. +3. **Post-cockroach bootstrap:** select the current-era snapshot/artifact and + API a consumer uses when the old era's ordered history is unavailable. +4. **Future offsets:** decide whether `from_offset > H` waits for the head or + fails with a typed ahead-of-head response. It does not affect the settled + `X`-means-next-input boundary. +5. **Access policy/limits:** decide whether replay remains + internal/network-restricted or needs application authentication, and set + rate limits before public exposure. + +Questions 2 and 3 shape step 3 and need Bart's review. Question 4 gates the WS +v2 contract. Question 5 gates public exposure. Question 1 is explicitly +non-blocking unless a consumer brings a concrete requirement. + +## 9. Review remarks (non-normative) + +- Cockroach recovery cannot currently supply historical user ops as ordered L2 + transactions from genesis. Therefore finalized replay stability is scoped to + an era. The new era retains the recovered application's absolute count as + `K`; a changed history version detects the boundary, and offsets below `K` + fail honestly rather than fabricating the missing feed. +- `EraId` replaces the earlier `instance_id` name because it describes an + externally visible history era, not a process or machine instance. It is + UUIDv4; the first-setup timestamp remains separate metadata. +- The reference scheduler count audit is landed: successful directs and user + ops share one checked typed boundary, overdue-direct ordering is preserved, + and `AppError` is fatal rather than skipped. The durable per-input mapping is + also landed; the public feed cutover is not. diff --git a/docs/plans/2026-07-track6-dump-api-design.md b/docs/plans/2026-07-track6-dump-api-design.md new file mode 100644 index 00000000..c599d263 --- /dev/null +++ b/docs/plans/2026-07-track6-dump-api-design.md @@ -0,0 +1,216 @@ +# Dump / `Application` API — Design Draft (Track 6) + +**Status: draft for review** (us, Bart). Companion to the Track 3 feed +design — review together; libdex's on-disk layout depends on both. On +acceptance, the trait contract graduates into +`docs/protocol/application-contract.md` and the lifecycle changes into +`docs/snapshots/lifecycle.md`. + +## 1. Motivation + +Three forces, one API: + +- **Cost.** `create_dump(&self)` is a full O(state) serialize + 3-fsync + ladder, run synchronously on the single lane thread at every batch close — + soft-confirmation acks stall for the duration. Tolerable for the toy + wallet; not for a multi-GB flat buffer. +- **Bart's app shape.** libdex runs natively against an mmap'd flat state + buffer — the Cartesi Machine's own storage approach (Bart implemented that + feature), whose ecosystem exploits CoW (`clone_stored`: reflink, plus + hardlinks — safe there only because stored images are immutable, see §10 — + with plain-copy fallback; ~2.6 ms vs ~350 ms full store at 533 MB in + Dave's measurements). +- **Verb review.** `create_dump / from_dump / delete_dump / + state_file_in_dump` is a leaky projection of what the lane needs; the + 2026-07 investigation mapped it against the CM/Dave verb set. + +Key investigation results this design builds on: the CM emulator has **no +commit/revert** — those are node-level orchestration over `clone_stored`, +and the sequencer already has both (DB row as commit point; older-dump + +replay as revert) correctly *off* the trait. The one missing primitive is +**cheap clone**. And the CM has since moved our way on durability: +machine-emulator PR #398 adds `rename_stored` and makes it and +`remove_stored` **durable (auto-synced)** — the commit-point idiom our dump +lifecycle hand-rolls. + +## 2. Requirements + +From the lane trace (R) and the wishlist (W): + +- **R1** Checkpoint the current state at batch close — atomic, + crash-durable *before* the DB row lands (invariant I13). +- **R2** Reconstruct at startup: latest checkpoint + replay. +- **R3** Dispose checkpoints (GC + orphan sweep). +- **R4** Serve canonical bytes over HTTP without instantiating the app; + bytes must equal the canonical machine's `inspect_state` output (the + watchdog byte-compares). +- **R5** Genesis construction (off-trait today, stays off-trait). +- **W1** Checkpoints cheap enough to not shape batch policy (CoW). +- **W2** Checkpointing off the ack path. +- **W3** Natural fit for an mmap'd-working-image app without penalizing + pure-RAM apps (WalletApp). + +## 3. The fork, decided: working-image model + +The investigation flagged one load-bearing fork: cheap clones require the +app to run against an on-disk image the lane can flush-then-clone (Dave's +`SHARING_ALL` model); `create_dump(&self)` serializing live RAM can never be +cheap. **This design takes the working-image model** — it is libdex's +natural shape, it is what the CM ecosystem optimizes, and WalletApp adapts +trivially (its "working image" is a file it rewrites on flush; cost +unchanged from today's serialize). + +## 4. Proposed trait + +```rust +pub trait Application: Send + Sized { + // --- execution surface unchanged --- + + /// Open the app on a working image directory. The sequencer owns the + /// directory's lifecycle; the app owns its contents. Called at startup + /// (from a cloned checkpoint) and after genesis materialization. + fn open(working: &Path) -> Result; + + /// Make the working image consistent and durable on disk: flush + /// app-level caches, msync mapped pages, fsync files. After `Ok`, the + /// on-disk image alone reconstructs this exact logical state via + /// `open`. Called by the lane at batch close, before cloning. + fn flush(&mut self) -> Result<(), AppError>; + + /// Clone the (flushed, not currently open) image at `from` into `to` + /// (must not exist). Default: recursive plain copy + fsync ladder. + /// CoW apps override with reflink/hardlink (FICLONE / clonefile), + /// keeping the same durability contract: on `Ok`, `to` survives an + /// immediate kernel crash. + fn clone_image(from: &Path, to: &Path) -> Result<(), AppError> { … } + + /// Delete an image directory the sequencer no longer references. + /// Default: remove_dir_all. + fn delete_image(prefix: &Path) -> Result<(), AppError> { … } + + /// Locate the single canonical state file inside an image without + /// opening the app. Contract unchanged from state_file_in_dump: + /// the file's bytes equal canonical `inspect_state` output (R4). + fn canonical_file_in_image(prefix: &Path) -> PathBuf; +} +``` + +Verb mapping: `open` ≈ CM `load(SHARING_ALL)`; `flush` ≈ the msync the CM's +dirty-page sidecars make optional; `clone_image` ≈ `cm_clone_stored`; +`delete_image` ≈ `remove_stored`. Commit stays sequencer-side (DB row; +sealing by durable rename per PR #398's precedent). Revert stays +sequencer-side (older checkpoint + replay). `from_dump`/`create_dump` +disappear: restore is `clone_image(checkpoint, working)` + `open(working)`; +checkpoint is `flush()` + `clone_image(working, checkpoint)`. + +## 5. Lane lifecycle changes + +Batch close becomes: `flush()` → `clone_image(working, staging)` → +sequencer writes `info.toml` + durable-renames staging into the dumps dir → +DB row in one tx (unchanged commit point). Filesystem-first ordering, the +"orphan dir possible, dangling row never" invariant, promotion, GC, leases, +and the whole `snapshot_dumps.rs` layer are **unchanged** — the storage half +is already representation-agnostic (opaque prefix keys). + +W2 (off-ack-path) falls out for CoW apps: `flush` + reflink is +milliseconds, and the expensive part (page write-back) is the kernel's +business afterward. A dedicated async stage is *not* designed in; if a +non-CoW app's flush is slow, that is the app's cost to fix by adopting CoW. + +Startup (R2): clone the promoted/pending checkpoint into a fresh working +dir, `open`, replay. The working dir is disposable state — never promoted, +never served, deleted on clean start. + +## 6. Crash-safety posture (unchanged, now sharper) + +I13 stays: nothing may reach the DB before the corresponding image is +durable. The split is now explicit: the **app** guarantees durability of +image *contents* (`flush`, `clone_image`); the **sequencer** guarantees +durability of *directory structure* (rename + dir-fsync — exactly what CM +PR #398 now bakes into `rename_stored`/`remove_stored`, validating the +posture). Tell Bart directly: the CM's historical no-fsync stance does not +apply here — a CoW `clone_image` override must fsync what reflink leaves +unsynced, and #398 shows the CM itself now agrees for the rename/remove +verbs. + +## 7. Serving (R4) and the libdex layout constraint + +`canonical_file_in_image` keeps the single-canonical-file contract — the +HTTP snapshot routes, lease protocol, and watchdog byte-compare all survive +untouched. The constraint to put in front of Bart *before* libdex's layout +freezes: the served file must byte-match canonical `inspect_state` output, +so either (a) the flat buffer's layout is itself canonical — fully +normalized, no allocator padding, no free-lists, no pointer-valued fields, +no uninitialized gaps — and the buffer file doubles as the canonical file; +or (b) libdex writes a separate canonical projection during `flush`, which +reintroduces O(state) serialize cost and partly defeats CoW. (a) is the +performant answer and a real design constraint on his buffer format. + +## 8. Migration & cleanups folded in + +- **WalletApp:** `open` mmap-or-reads its file; `flush` = today's + serialize+fsync ladder; defaults cover the rest. No capability lost. +- **Genesis:** concrete-type constructor materializes the initial working + image, then the normal flush/clone path checkpoints it (R5 unchanged). +- **`SafeInputRecord` shim** (`storage/l1_inputs.rs`): collapse + `StoredSafeInput`/`IngestedSafeInput` into one honest row model in a dedicated + cleanup. The provenance/clock decision below is settled; it no longer blocks + that cleanup. +- **Direct-input clock semantics (settled with Track 3):** `block_timestamp` + is persisted and served as feed provenance only; it is not an application + transition input. Directs execute at their exact L1 inclusion block and user + ops at their frame's safe block, as owned by the + [`Application` contract](../protocol/application-contract.md#3-the-safe-block-clock--last_executed_safe_block). + No timestamp-bearing trait change is needed, and this no longer blocks + libdex's state design. + +## 9. Open questions + +1. **Working-image locking:** the CM uses flock to make `SHARING_ALL` + exclusive. Do we require the app to hold an equivalent lock, or does the + sequencer's single-lane discipline suffice? (Lean: sequencer discipline + suffices; a lock is cheap defense — app's choice.) +2. **`flush` durability scope:** must `flush` fsync, or is fsync deferred to + `clone_image`? (Lean: `clone_image` owns durability of the *clone*; + `flush` owns consistency of the *source* — msync yes, fsync optional.) +3. **Non-reflink filesystems:** plain-copy fallback makes checkpoint cost + O(state) again silently. Log loudly at startup when the dumps dir does + not support reflink? (Lean: yes — one probe at bootstrap.) +4. **Dirty-page tracking for hashing/inspect:** the CM's `.dpt` sidecars + make post-clone hashing touch only dirty pages. libdex would need its + own write-barrier to replicate; out of scope for the sequencer API but + worth flagging to Bart as a cost driver for (a) in §7. + +## 10. Review remarks (non-normative, open design) + +These remarks record issues raised during review; they do not replace the +proposed trait or lifecycle above. (An earlier revision promoted the first +remark into §4's `clone_image` contract and misattributed that promotion to +a maintainer instruction; the 2026-08-01 review corrected the record and the +remarks-only posture is restored. Whether hardlinks stay in §4's suggestion +is settled in design review with Bart, alongside the rest of the trait.) + +- Hardlinks are not a valid implementation of the mutable-working-image to + immutable-checkpoint clone. Later in-place or mmap writes through either + path mutate the same inode and therefore the checkpoint. Any accepted + implementation should require a real CoW clone (`FICLONE`/`clonefile`) or + an independent copy, with a test that mutating the reopened working image + cannot change checkpoint bytes. +- Synchronous durability is likely simple and fast enough, and should remain + the baseline while it is measured. Measure the relevant phases separately: + app flush/msync, source synchronization if required, clone or copy, + destination data and metadata synchronization, staging rename, parent + directory synchronization, and DB commit. Measurements should cover + representative state sizes and dirty-page ratios and report tail latency, + not only a best-case reflink time. +- The contract that `clone_image -> Ok` survives an immediate crash is + stronger than the statement in §5 that expensive page writeback may remain + the kernel's work afterward. Until filesystem-specific ordering and sync + requirements demonstrate both claims together, deferred writeback is an + unresolved durability issue rather than work safely removed from the + checkpoint path. +- An asynchronous checkpoint stage need not be designed preemptively. If the + synchronous phase measurements meet the latency budget, keeping the + durability boundary synchronous is preferable. Async staging should be + reconsidered only if measurements show that the required flush and sync + operations materially violate that budget. diff --git a/docs/plans/2026-08-authority-boundary-adr.md b/docs/plans/2026-08-authority-boundary-adr.md new file mode 100644 index 00000000..2ff8738b --- /dev/null +++ b/docs/plans/2026-08-authority-boundary-adr.md @@ -0,0 +1,208 @@ +# ADR: The Runtime Authority Boundary + +The architecture decision record for how authority — over speculative state, +promises, process lifetime, and recovery admission — is owned in the +sequencer. The mechanisms below are landed; the decision history and the +review trail that shaped them live in +[`../review/register.md`](../review/register.md) and the dated ledgers beside +it. + +## Context + +Repeated review rounds found instances of one bug class: an effect, mutation, +or admission point forgot to consult a distributed terminal predicate. The +root cause was not one missing check — authority was implicit across workers, +markers, triggers, classifiers, and shutdown guards. + +The policy separates into three guarantees: + +1. **G1 — in-process closure:** after terminal closure, no new + authority-bearing work is accepted. +2. **G2 — no silent fast-path:** an interrupted or terminal run cannot skip + inspection on the next boot. This is carried by the unconditional recovery + reducer: every boot re-derives its decisions from durable facts, never + from the previous process's verdict. +3. **G3 — scoped divergence freeze:** once the canonical-divergence fact is + committed, the accepted frontier, batch tree, and snapshot promotion are + frozen immediately ([I15](../invariants.md)). `DangerDetector` owns prompt + process shutdown; the running inclusion lane also refuses + opportunistically when its existing frontier read observes the poison. + User-op work authorized before runtime observation may still commit and + acknowledge. + +The enforceable unified policy: + +> After in-process terminal closure linearizes, no new **authority-bearing +> mutation or promise** may be authorized. Work already accepted may +> complete. A durable divergence fact freezes its persisted acceptance +> domain immediately. It is deliberately not a per-user-op transaction +> fence. Containment/telemetry writes and immutable operator reads are not +> authority-bearing. A later command may start only after exclusive process +> ownership and the admission facts pass. + +An acknowledgement, live feed event, nonce reservation, or L1 submission is +authority-bearing. Streaming an already-immutable operator snapshot is not. +Absolute cancellation of an effect already handed to the network is neither +claimed nor implementable; the zombie-transaction model already accounts for +that case. + +## The mechanisms + +Four mechanisms, each solving a different problem: + +### 1. `RuntimeScope`: structured process ownership + +Every command acquires the OS-held exclusive data-directory lock before +inspection (`runtime/process_lock.rs`). For `run`, ownership transfers into a +`RuntimeScope` shared by all runtime-owned data-directory work: the lock, +terminal abort watchdog, containment authority, and fault recorder in one +capability, constructible only from a held lock. The pure notification half +is the slim `ShutdownSignal`. `RuntimeScope::authorize()` mints the +borrow-scoped `Authorized` token that the externalization primitives (ack, +L1 send, WS emit, snapshot-stream start) require — forgetting the +containment consult is a compile error, not a convention. + +The lock is released only after every runtime-owned child has actually +stopped; a dropped `JoinHandle` detaches rather than stops, so each worker +and nested blocking task retains its own capability clone until its closure +really ends. This is a cheap local foot-gun guard, not distributed fencing: +it prevents two processes on one data directory and nothing more. + +Runtime construction is prepare → admit → launch: every fallible or awaited +operation happens while zero tasks exist; final admission re-runs the +recovery reducer over one consistent fact set; launch spawns every worker in +one infallible, non-yielding block, consuming the single-use +`RuntimeAdmission` witness. A preparation failure cannot leave a partially +launched runtime, and no refusal or retry can mint the witness. + +### 2. Fact-derived admission and the terminal-fault black box + +Admission is governed by three facts, each with one owner: the kernel +process lock (concurrent owners), two-sided `setup_complete` (command +ordering), and `canonical_divergence` (the one absorbing refusal — only a +fresh-directory cockroach rebuild proceeds). There is no lifecycle admission +state machine and no operator acknowledgement: standard recovery is +automatic, and restart policy after a terminal fault is the exit-code +contract (30 = do not restart, page), enforced by the supervisor. The only +durable telemetry is the `terminal_faults` black box — append-only +terminal-cause rows, written best-effort and verdict-neutrally; nothing +reads it for decisions. + +The accepted trade, eyes open: a known-terminal fault refuses at +re-detection rather than at a boot gate. Every fault whose evidence the boot +path reads re-refuses before the first soft confirmation; the narrow +residual window is recorded in the threat model, and the honesty backstops +(rollbackable soft confirmations, the watchdog byte-compare, the divergence +freeze) never depended on a boot gate. + +### 3. Recovery as a pure run reducer + +Normal `run` startup is one unconditional loop: + +```text +inspect -> classify once -> decide -> perform at most one phase -> inspect again +``` + +`reduce_recovery` is pure policy over one transactionally consistent +`RecoveryInspection` plus boot-local phase progress. Local absorbing facts +are inspected before any provider call, so a transient RPC error can never +mask a persisted divergence. Closed recovery is phase-granular — +`Flush → inspect → Sync → inspect → Cascade → inspect` — with the flush's +safe-block observation carried as an ephemeral, memory-only witness: a crash +loses it and the next boot repeats the idempotent flush. There is +deliberately no durable recovery-phase state machine. The +[`admission.tla`](../recovery/admission.tla) model verifies the controller +ordering; [`docs/recovery/README.md`](../recovery/README.md) owns the design. + +Setup/rebuild, maintenance flush, and normal-run recovery retain distinct +typed controllers: their facts are unrelated, and a generic command reducer +would enlarge the state machine without closing an enforcement hole. + +### 4. SQLite-centered runtime and the two-regime inclusion lane + +SQLite is the durable coordination boundary between components. The input +reader atomically commits `safe_inputs`, `l1_safe_head`, +`safe_accepted_batches`, and any `canonical_divergence` fact in one sync +transaction; the lane reads that durable projection. The one deliberate +exception is HTTP ingress ↔ inclusion lane (bounded MPSC + oneshot), because +low-latency request/response over the lane's in-memory application is +unwieldy through SQLite — an exception for one local interaction, not a +precedent for an in-memory component bus. + +The lane has two regimes. The **fast user-op regime** dequeues at most one +bounded chunk per turn — accepted or rejected — and commits the accepted +subset at most once with `synchronous=FULL`; only that commit authorizes +acknowledgements. Making the dequeue chunk itself the turn boundary keeps +entry to reconciliation independent of acceptance outcome, so rejected +floods cannot starve the frontier check. The **L1 reconciliation regime** +fires when the observed safe head is at least five blocks past the open +frame's clock: it consumes the complete accumulated newly-safe range, +promotes at most once, and opens exactly one frame at the observed tip — +jumps are never interpolated. There is no elapsed-time budget, preemption, +or resumable partial cursor inside a turn: the supported deployment assumes +the application promptly digests the whole range (revisit only if production +measurements disprove that). + +Authority remains role-local and auditable: a FULL-committed user-op chunk +authorizes its acknowledgement; a valid sealed batch plus the durable +write-before-broadcast watermark authorizes an L1 submission; committed +valid rows plus their canonical `executed_inputs` attribution authorize feed +output. An already-authorized effect may finish after a later terminal +transition. + +## Rejected alternatives (do not re-propose without new evidence) + +- **`RunEpoch`** (a globally threaded internal fencing epoch): the OS lock + plus structured task lifetime plus fresh per-scope channels already make + an old sender unable to reach a new receiver, and there is no in-process + hot restart to fence against. Revisit only if in-process restart or + multiple admitted runtimes under one lock are introduced. +- **`EffectGate` / `LiveKernel`** (a universal effect mutex or actor): would + duplicate the role-local linearization points the system already needs and + force the reader and latency-critical lane through a new in-memory + authority protocol, adding a second state machine without making the + narrow content-identity check a complete divergence oracle. The + `Authorized` token is not this: no mutex, no actor, no runtime state — + the same predicate moved into the signatures of the operations it guards. +- **A generic command controller** over setup/rebuild/run/maintenance: + their facts are unrelated; combining them enlarges the cross-product state + machine without closing an enforcement hole. +- **A per-user-op divergence query** (or reader mailbox) on the hot path: + the content-identity check is complete only for accepted-batch content + identity ([I9](../invariants.md)); paying a per-chunk query would not buy + a complete safety boundary. +- **A durable recovery-phase ledger**: the flush witness is boot-local by + design; persisting it would re-create a state machine whose only effect is + skipping an idempotent flush. +- **A durable boot gate on terminal verdicts**: a gate on a non-fact needs + an operator acknowledgement to exit, and the acknowledgement carries no + information the fact-derived reducer doesn't already re-derive. + +## External history + +```text +HistoryVersion = (EraId, RecoveryGeneration) +HistoryPosition = (HistoryVersion, ExecutedInputCount) +``` + +`EraId` (UUIDv4, minted write-once in the baseline transaction) identifies a +setup/rebuild era; `RecoveryGeneration` increments exactly once in the +standard-recovery transaction iff it invalidates at least one valid batch; a +clean restart changes neither. The pair is an equality/discontinuity token, +not an ordered counter. The feed coordinate is the canonical +`Application::executed_input_count()`, never a SQLite cursor. The durable +foundation is landed ([I18](../invariants.md), [I20](../invariants.md)); +the public wire projection is owned by the +[Track 3 handoff](2026-07-track3-feed-replay-design.md#7-ordered-implementation-handoff). + +## Performance posture + +The product contract is `POST /tx` acknowledgement under 500 ms. Same-host +release sweeps across the cutover found no material regression: ACK p99 at +or below ~50 ms through concurrency 256 with zero rejections, concurrency-1 +HTTP ACK p50 around 13 ms (submit-to-matching-WS-event p50 roughly double — +name which metric "round-trip" means). Same-host numbers are method-specific +regression evidence, never capacity claims: at high concurrency the load +clients contend with the sequencer, so the plateau is machine saturation. A +separate-machine load generator is required for capacity measurement, and +round-trip remeasurement belongs with the public history/API projection. diff --git a/docs/protocol/application-contract.md b/docs/protocol/application-contract.md index 2ff8a184..e93a8b25 100644 --- a/docs/protocol/application-contract.md +++ b/docs/protocol/application-contract.md @@ -2,11 +2,12 @@ The FFI seam. An app plugs into the sequencer by implementing [`Application`](../../sequencer-core/src/application/mod.rs). The sequencer -assumes the contracts below **without runtime enforcement** — it links the app, -calls it on the hot path and during catch-up, and trusts it to be a pure -deterministic state machine. A violation is not caught; it surfaces as -scheduler/sequencer divergence, which under rollup semantics is -theft-equivalent ([threat model](../threat-model/README.md), "Self-trust"). +links the app on the hot path and during catch-up, and trusts it to be a pure +deterministic state machine. The shared execution boundary enforces the +scheduler-owned progress transition; application-specific determinism and +mutation semantics remain self-trusted. A violation is fatal because silent +scheduler/sequencer divergence is theft-equivalent under rollup semantics +([threat model](../threat-model/README.md), "Self-trust"). This document **owns** the contract. [`AGENTS.md`](../../AGENTS.md) §"Application Trait Contract" is the map. The placeholder wallet @@ -17,11 +18,11 @@ production app will wrap a Cartesi Machine behind the same trait. ## The execution methods -| Method | Mutates? | Clock advance | Failure contract | +| Method | Mutates? | Progress effect | Failure contract | |---|---|---|---| -| `validate_user_op(sender, op, current_fee) -> Result<(), InvalidReason>` | **No** — pure, read-only | — | `Err(InvalidReason)` ⇒ op skipped, no state change | -| `execute_valid_user_op(valid, safe_block) -> Result` | Yes | `clock = max(clock, safe_block)` | `Internal` is **fatal** (see *Replay safety*) | -| `execute_direct_input(input) -> Result` | Yes | `clock = max(clock, input.block_number)` | `Internal` is **fatal** | +| `validate_user_op(sender, op, current_fee) -> Result<(), InvalidReason>` | **No** — pure, read-only | unchanged | `Err(InvalidReason)` ⇒ op skipped, no state change | +| `apply_valid_user_op(capability, valid, safe_block) -> Result` | application state only | the shared `execute_valid_user_op` commits count `+1` and `clock = max(clock, safe_block)` after `Ok` | any `AppError` is **fatal** (see *Replay safety*) | +| `apply_direct_input(capability, input) -> Result` | application state only | the shared `execute_direct_input` commits count `+1` and `clock = max(clock, input.block_number)` after `Ok` | any `AppError` is **fatal** | `MAX_METHOD_PAYLOAD_BYTES` is the app's declared upper bound on a method payload's encoded size (selector + args). The sequencer treats it as a sizing @@ -33,17 +34,27 @@ payload larger than it declares. ### One execution entry point -User ops are **never** executed by calling `execute_valid_user_op` directly. -They go through the free function +Application hooks are never called directly. User ops go through the free +function [`validate_and_execute_user_op`](../../sequencer-core/src/application/mod.rs), which enforces the protocol guard `max_fee ≥ current_fee` *before* app -validation, then calls `validate_user_op`, then `execute_valid_user_op`. It is a -free function — not an overridable trait method — precisely so no impl can skip -the guard. Both consumers (the inclusion lane and the canonical scheduler) call -it; that shared call path is half of the +validation, then calls the shared `execute_valid_user_op`; directs go through +the shared `execute_direct_input`. Those two functions stage and commit +`ApplicationProgress` around the app's `apply_*` hook. The raw hooks require a +borrowed opaque apply capability, and mutable progress access requires a +separate borrowed opaque commit capability; only the shared boundary can +construct either one. A caller therefore cannot invoke a raw hook or mutate +count/clock directly. The inclusion lane, catch-up, recovery fold, and +canonical scheduler all use these paths; that shared boundary is half of the [duality](scheduler-semantics.md#the-three-implementations-and-why-they-agree) agreement. +The boundary checks that progress remains unchanged after validation and after +an application hook, on both `Ok` and `Err`. After a successful hook it commits +the precomputed successor and re-reads the immutable getter to assert that the +application's getter/mutator pair is coherent. Count exhaustion is checked +before the hook runs. + Consequently `validate_user_op` must **not** re-implement the `max_fee` guard as its contract (the free function already owns it); it checks only app-level predicates — nonce match for user-op replay protection, and fee-balance @@ -62,7 +73,7 @@ Execution must be a pure function of `(input, current state)`: point, threads, or any other nondeterminism in a consensus path. - `validate_user_op` is **pure and read-only** — no mutation, no time dependence, no randomness. State changes flow *exclusively* through the two - execute methods. Mutating from `validate_user_op` breaks replay (validation + `apply_*` hooks. Mutating from `validate_user_op` breaks replay (validation runs on a different schedule than execution). - The same bytes against the same state must always produce the same outcome and the same `AppOutputs`. This is what lets the off-chain mirror predict the @@ -75,23 +86,39 @@ The sequencer persists every executed input and, on restart, replays them in order against a fresh instance to rebuild state (catch-up). Therefore: - **Any input that executed successfully live must execute successfully on - replay.** Catch-up treats `AppError::Internal` as **fatal** — it aborts - startup and the sequencer cannot resume. Never return `Internal` for a byte - sequence that previously succeeded. + replay.** Catch-up treats every `AppError` as **fatal** — `Internal` and + `Io` alike (every caller propagates both identically) — it aborts startup + and the sequencer cannot resume. Never return an error for a byte sequence + that previously succeeded. - Prefer `ExecutionOutcome::Invalid` for malformed or ill-typed input caught at the app level — `Invalid` is replay-safe (it deterministically skips, live and on replay). Reserve `AppError::Internal` for genuine invariant violations ("validated user op cannot pay fee") — real bugs, deliberately fatal, not adversarial inputs. +- `AppError` defines no canonical successor. Application-specific mutation is + not rolled back when a hook fails; every production caller terminates that + execution path and discards the instance rather than continuing from it. ### 3. The safe-block clock — `last_executed_safe_block` -`last_executed_safe_block() -> u64` returns the **maximum block carried by any -input this instance has executed** (frame `safe_block` for user ops, L1 +`last_executed_safe_block() -> u64` reads the safe-block field of the embedded +`ApplicationProgress`: the **maximum block carried by any input this instance +has executed** (frame `safe_block` for user ops, L1 `inclusion_block` for directs), or 0 if nothing has executed. -- It is **carried in execution, not a setter** — every execute method advances - it via `max`, so an app cannot execute an input and forget to move the clock. +The live sequencer advances its frame clock on a best-effort safe-block policy: +once the observed safe head is at least five blocks beyond the open frame, it +opens exactly one frame at the observed tip. A delayed or epoch-sized head jump +is not interpolated. All user ops in that frame execute sequentially with the +same logical block value; newly covered directs execute first but retain their +own exact L1 inclusion blocks. A clock-only empty frame does not call the +application or autonomously advance this method—it supplies a newer clock to a +later executed user op. + +- It is **scheduler-owned, not a setter** — the shared execution boundary + advances it via `max` only after the application hook returns `Ok`. +- Count zero implies clock zero: no input has executed from which a non-zero + clock could have been derived. - It **must survive `create_dump`/`from_dump` round-trips** (it is part of the logical state a dump captures). - Recovery reads it as `A`, the safe block a checkpoint state reflects, and the @@ -99,10 +126,57 @@ input this instance has executed** (frame `safe_block` for user ops, L1 ([cockroach recovery](../recovery/cockroach.md)). A wrong clock mis-defines that range. -`executed_input_count() -> u64` is a diagnostic seam — replay/catch-up and the -snapshot byte-comparison compare a live instance against a replayed one with it. - -### 4. Dump lifecycle round-trip +### 4. Canonical history cursor: `executed_input_count` + +`executed_input_count() -> ExecutedInputCount` is the authoritative boundary +coordinate of application execution, not a diagnostic counter. It starts at +zero. Each successful shared `execute_valid_user_op` or `execute_direct_input` +call advances it by **exactly one**; validation failures, rejected user ops, +inputs merely queued for later execution, and any other unexecuted input leave +it unchanged. An `AppError` is fatal and does not define a canonical successor +state. The boundary checks the `u64` successor before calling the application, +so exhaustion fails before application mutation; wrapping and saturating +arithmetic are unavailable on the newtype. + +The count names the next history entry the application is ready to execute. If +an application is at count `X`, history input `X` is the input that must move it +to count `X + 1`. A subscriber holding that application state therefore resumes +from offset `X`; it must not translate between an application count and a +separate feed position. + +The count is logical application state and **must survive +`create_dump`/`from_dump` round-trips**. Replay and both recovery procedures +must reconstruct the value implied by the resulting application state: a +standard recovery may roll it back to the retained prefix and advance it over +replacement canonical inputs, while a cockroach-recovered dump supplies the +absolute count from which the newly available history continues. + +> **Cutover status:** the typed `ApplicationProgress` execution boundary, +> scheduler transition audit, placeholder application's durable count, +> per-input SQLite attribution, and snapshot/catch-up agreement checks are +> landed. The current feed still exposes a SQLite-rowid `from_offset`; the +> history-version and canonical-offset HTTP/WS projection remain Track 3 work. +> Until that API cutover, clients must follow the README. + +### 5. Operational capacity for L1 reconciliation + +A supported production application must promptly execute the complete +accumulated input range the persisted frontier can expose in one L1 +reconciliation turn, including catch-up/backlog within the supported operating +envelope. Once the lane enters that turn, it processes the whole range before +returning to user-op work. The sequencer deliberately provides no elapsed-time +cutoff, preemption, or durable timeout-and-resume cursor for application +execution. Scratch paging may bound memory or read-query size; the +drain/promotion commit remains atomic and the logical turn is not resumable. + +This is an explicit deployment assumption, not a deterministic state-transition +rule. A request that overlaps reconciliation may see additional acknowledgement +latency. Revisit the scheduling design only if application cost, L1 +capacity/finality/catch-up behavior, or measurements show that the complete +newly-safe range is not promptly digestible. Do not add a second scheduler +speculatively. + +### 6. Dump lifecycle round-trip The snapshot lifecycle ([snapshots](../snapshots/lifecycle.md)) drives four dump methods. `Self`-typed methods load/construct; the associated functions are @@ -141,9 +215,10 @@ lives on the concrete type, called by the runtime at bootstrap. | Purity / determinism | the [duality](scheduler-semantics.md) (off-chain prediction = canonical fold); recovery `fold_replay` | | Replay safety (`Internal` fatal) | catch-up on every restart | | Safe-block clock survives dumps | cockroach recovery's `A`; snapshot offset accounting | +| Executed-input count survives dumps | canonical history offsets; replay; standard and cockroach recovery; planned Track 3 subscription continuity | | `create_dump` in-method fsync | crash-safety of the dump/row ordering ([I13](../invariants.md)) | | `state_file_in_dump` = canonical bytes | the watchdog / indexers reading finalized state | -| One execution entry point | the `max_fee` protocol guard's non-bypassability | +| One execution entry point | non-bypassable `max_fee` and count/clock transitions | ## Rejection semantics every app implements @@ -157,5 +232,5 @@ mutation and are not persisted**: it tracks DA, not compute. Deposits are **direct-input-only** (L1 → L2) and must never be represented as -user ops; that is why `execute_direct_input` is required (no default) — a no-op -default would silently strand every deposit. +user ops; that is why the `apply_direct_input` hook is required (no default) — +a no-op default would silently strand every deposit. diff --git a/docs/protocol/scheduler-semantics.md b/docs/protocol/scheduler-semantics.md index f9a31eec..9acb57cb 100644 --- a/docs/protocol/scheduler-semantics.md +++ b/docs/protocol/scheduler-semantics.md @@ -11,7 +11,9 @@ This document **owns** that algorithm. [`AGENTS.md`](../../AGENTS.md) §"Sequenc Scheduler Duality" is the map; this is the detail. The reference implementation is [`Scheduler`](../../sequencer-core/src/scheduler/mod.rs) — the same source compiled into the on-chain canonical machine and run bare-metal by the recovery -fold, so the two targets agree *by construction*. +fold, so those two targets agree *by construction*. This prose is the intended +protocol contract; the reference implementation embodies it but remains code +that can contain bugs and require hardening. --- @@ -27,7 +29,8 @@ address, never by a tag byte** ([`process_input`](../../sequencer-core/src/sched | anything else | a **direct input** (deposit) | appended to the *fridge* (the direct-input FIFO), drained later | The payload is opaque to classification; app-specific decoding happens inside -`Application::execute_direct_input` (see the +the capability-gated `Application::apply_direct_input` hook, reached through +the shared `execute_direct_input` boundary (see the [application contract](application-contract.md)). --- @@ -106,11 +109,77 @@ For each frame, in order ([`process_batch_payload`](../../sequencer-core/src/sch guard fails — the fold is a pure deterministic function and emits no diagnostics at the library seam. +An `AppError` from either execution path is fatal: there is no canonical +successor to the partially evaluated input, so the canonical harness, +inclusion lane, catch-up, and recovery fold all stop rather than continue. + This ordering — directs ≤ `S_K` then ops validated on top of them — is exactly what the inclusion lane writes into frame K's wire content -([I2](../invariants.md#i2-drain-attribution-drained-directs-land-in-the-new-frame)), +([I2](../invariants.md#i2-drain-attribution-accumulated-directs-land-in-the-clock-advanced-frame)), which is what keeps the off-chain prediction faithful. +### Sequencer frame-clock policy + +The scheduler constrains frame clocks to be non-decreasing and no later than +the batch inclusion block; it does not require a frame for every L1 block. +During an admitted live run, the sequencer advances logical frame time when the +latest persisted safe head `H` is at least five blocks beyond the open frame +clock `S`. It then opens exactly one frame at `H`, drains the complete +newly-covered direct range, and uses `H` for subsequent user ops. An observation +jump from 100 to 132 therefore creates one frame at 132, not synthetic frames at +each missed five-block boundary. Intermediate empty frames would execute +nothing, so their omission is scheduler-equivalent. Bootstrap and recovery are +anchoring transitions rather than live clock ticks: they may open a fresh Tip +at their proven checkpoint/current safe head without applying the five-block +delta. + +Direct-input presence is not another rotation condition: the five-block tick +may have an empty direct prefix, while directs observed below the threshold +wait for the next tick and retain their own inclusion-block clock when +executed. Batch closure is orthogonal and necessarily creates the successor +batch's first frame at the unchanged `safe_block`; equal adjacent frame clocks +are valid. Thus block distance is the only reason logical frame time advances, +not the only reason a frame row exists. + +--- + +## The application-history offset + +The feed offset is semantically +[`Application::executed_input_count()`](application-contract.md#4-canonical-history-cursor-executed_input_count). +It starts at zero and names the next application input to execute. An +application at count `X` is ready for history entry `X`; successfully executing +that entry advances the application to `X + 1`. + +The scheduler determines which InputBox material becomes an executed +application input. Its count transitions are therefore: + +| Scheduler event | Application execution | Count effect | +|---|---|---| +| Direct arrives and is appended to the fridge | none yet | unchanged | +| Covered or overdue direct is successfully executed | `execute_direct_input` returns `Ok` | exactly `+1` | +| User op passes signature, protocol, and app validation and is successfully executed | `execute_valid_user_op` returns `Ok` | exactly `+1` | +| User op has an unrecoverable signature or is rejected by protocol/app validation | none | unchanged | +| Batch envelope is decoded, accepted, skipped, or rejected | none for the envelope itself; contained executions are counted by the rows above | unchanged for the envelope | +| Empty batch | none | unchanged | +| Either execution method returns `AppError` | fatal invariant failure | no canonical successor is defined | + +The censorship backstop still runs before classification, so a malformed, +stale, or otherwise rejected batch can indirectly advance the count by causing +overdue directs to execute first. The batch itself never contributes an entry. + +This coordinate must agree across the canonical fold, the inclusion lane, +catch-up/replay, and recovery. Standard recovery may replace an invalidated +suffix at the same application offsets; cockroach recovery resumes from the +absolute count persisted in the recovered application state even when older +history is no longer locally available. + +> **Cutover status:** the typed execution boundary, scheduler count +> transitions, and durable per-input mapping are landed. Physical +> `sequenced_l2_txs.offset` remains SQLite rowid and the existing WebSocket +> still exposes that cursor; changing the public protocol to canonical offsets +> and `HistoryVersion` remains Track 3 work. + --- ## The three implementations (and why they agree) @@ -138,12 +207,22 @@ I1 names three places this algorithm lives. They are not three rewrites; two are self-bug (a fault state to crash on), not an adversarial input to predict. This document is the cross-reference home for the omission — the asymmetry is intentional, not a missing check. +- **The content-identity check is complete relative to #2, not an independent oracle for #1.** For + every at/above-anchor landing that `scheduler_accepts` accepts, the frontier + builder exhaustively finds a byte-identical valid local closed batch + (`Match`), no local batch (`Foreign`), or different bytes (`Mismatch`); the + latter two durably freeze the frontier. Because the check shares #2 and its + structural omissions, it cannot prove that #1, application state, or trusted + collapsed history is correct. A structurally malformed foreign landing may + conservatively record divergence even if #1 would reject it; that false + positive is accepted under the sequencer self-trust model. See I9/I15 for the + runtime and recovery boundary. - **The expected-nonce fold** is homed once, next to `scheduler_accepts`, as [`advance_expected_batch_nonce`](../../sequencer-core/src/protocol.rs). The submitter's `decide_submit_start` consumes it directly. The frontier builder `populate_safe_accepted_batches` keeps a deliberate inline copy of the same advance — its loop interleaves the advance with two storage-only side effects - (the R2 content-identity check and the `canonical_divergence` freeze) that + (the content-identity check and the `canonical_divergence` freeze) that cannot move below the protocol layer; sharing the fold there would force a callback contract. The duplication is intentional and documented at the call site. @@ -161,12 +240,15 @@ the agreement — only review and the duality tests. **Change one, check all.** - [I1](../invariants.md#i1-scheduler-acceptance-semantics-agree-across-all-implementations) — the three implementations agree (this document is its prose). -- [I2](../invariants.md#i2-drain-attribution-drained-directs-land-in-the-new-frame) +- [I2](../invariants.md#i2-drain-attribution-accumulated-directs-land-in-the-clock-advanced-frame) — drained directs land in the new frame, so the lane's wire content matches the fold's drain-before-ops order. - [I3](../invariants.md#i3-frame-safe_blocks-are-non-decreasing-along-the-spine) — frame `safe_block`s are non-decreasing, the lane-side mirror of gate d's monotonicity rule. +- [I19](../invariants.md#i19-application-progress-advances-only-at-the-shared-execution-boundary) + — every successful application input advances one typed count/clock pair; + rejected inputs do not, and `AppError` is terminal. ## Test-pinned properties @@ -191,3 +273,28 @@ The duality's load-bearing edge cases each have at least one test (`non_monotonic_safe_blocks_invalidate_batch`, `frame_safe_block_above_inclusion_block_invalidates_batch`, `wrong_batch_nonce_is_rejected_without_consuming_nonce`). + +--- + +## Reference-scheduler count discipline + +Every canonical input crosses the same typed execution +boundary in the scheduler, live lane, catch-up, and recovery fold: + +1. Successful directs and validated user ops return their pre-execution + `ExecutedInputCount` and advance exactly once with checked arithmetic. + Rejection, bad signatures, envelopes, empty batches, and stale/structural + skips leave it unchanged. +2. `AppError` is fatal everywhere; there is no parallel scheduler-only + transition after a possibly partial hook failure. +3. Distinct opaque capabilities let application hooks mutate application state + without granting them authority to overwrite scheduler-owned progress. The + shared boundary checks progress before/after both successful and failing + hooks and asserts getter coherence after commit. +4. Tests pin direct/user advancement, every skip/reject family, pre-batch + overdue-drain ordering, overflow, dump round-trips, nonzero recovery bases, + and live/replay mapping agreement. + +Additional scheduler changes may still be batched separately, but they must +preserve the transition table above and the shared boundary rather than +reintroducing a scheduler-local count. diff --git a/docs/recovery/README.md b/docs/recovery/README.md index 6114124e..028e279d 100644 --- a/docs/recovery/README.md +++ b/docs/recovery/README.md @@ -1,6 +1,6 @@ # Batch Recovery -This document describes the recovery design for the sequencer: how the system detects that batches are failing to land on L1, and how it recovers to a consistent state. The design is verified with bounded TLA+ model checking ([`preemptive.tla`](preemptive.tla)). +This document describes the recovery design for the sequencer: how the system detects that batches are failing to land on L1, how startup recovers to a consistent state, and where runtime authority begins. Two complementary bounded TLA+ models cover the design: [`preemptive.tla`](preemptive.tla) for batch/slot safety and [`admission.tla`](admission.tla) for startup phase ordering and admission. They do not currently model the external era/generation/base metadata or the derived canonical `executed_inputs` projection; their crash atomicity is enforced by the SQLite transaction boundaries and schema triggers described below. See `AGENTS.md` "Batch Staleness and Recovery" for quick-reference tables and function names. @@ -8,21 +8,23 @@ See `AGENTS.md` "Batch Staleness and Recovery" for quick-reference tables and fu The sequencer's recovery loop spans two process lifetimes: -1. **In-process detection.** The `DangerDetector` polls `Storage::check_danger` on a cadence. When any non-`Safe` status fires (`CanonicalDivergence`, `L1ViewStale`, `ClosedBatchInDanger`, `TipInDanger`, or `EstimatedBatchInDanger`), the runtime converts that into `DangerDetectorExit::DangerDetected` under `RunError::Worker` and the process exits with non-zero status. +1. **In-process detection.** The `DangerDetector` polls `Storage::check_danger` on a cadence. When any non-`Safe` status fires (`CanonicalDivergence`, `L1ViewStale`, `ClosedBatchInDanger`, `TipInDanger`, or `EstimatedBatchInDanger`), the runtime converts that into `DangerDetectorExit::DangerDetected` under `CommandError::Worker`, closes intake, and drains the workers before returning a non-zero status. Canonical divergence is terminal and arms the independent two-second abort bound; expected-recovery and retryable arms remain cooperatively graceful. 2. **External respawn.** An orchestrator (systemd, k8s, …) restarts the process. -3. **Startup dispatch.** The fresh boot runs `run_preemptive_recovery` before any writers come online: sync L1, re-run `check_danger`, then `decide_startup_action` routes to one of `Proceed`, `RecoverTip`, `FlushAndCascade`, or `Refuse`. Recovery actions run their DB mutations as single SQLite transactions; `Proceed` intentionally does no DB writes. +3. **Startup reducer.** The fresh boot reads divergence, danger, finalized-snapshot presence, Tip presence, and the safe head in one local transaction. The pure reducer selects at most one phase. Every completed phase returns to local inspection before another phase or admission. Initial Sync is itself a phase, so an already-persisted divergence refuses before the first provider call. +4. **Prepare, admit, launch.** A clean decision permits task-free, fallible runtime preparation. Startup then invokes the same reducer once more over one consistent fact set, mints the single-use `RuntimeAdmission` witness, and consumes it in an infallible, non-yielding worker launch. The detector trip and the startup dispatch share the same `check_danger` function; the detector cares only that *some* arm fired, while the startup dispatch examines *which* arm fired to pick the right action. Key abstractions, by responsibility: -- **`DangerDetector`** ([`recovery/detector.rs`](../../sequencer/src/recovery/detector.rs)): tiny background task that calls `Storage::check_danger` on a cadence. Never writes to the DB, never talks to L1. Exits with `DetectorExit::RecoveryRequired` when any non-`Safe` status fires. The runtime converts that into a `DangerDetectorExit::DangerDetected` worker exit and the process exits. The dispatch difference between statuses only matters at the next startup, where `decide_startup_action` re-runs `check_danger` and routes accordingly. +- **`DangerDetector`** ([`recovery/detector.rs`](../../sequencer/src/recovery/detector.rs)): tiny background task that calls `Storage::check_danger` on a cadence. Never writes to the DB, never talks to L1. Exits with `DetectorExit::RecoveryRequired` when any non-`Safe` status fires. The runtime converts that into a `DangerDetectorExit::DangerDetected` worker exit, requests process-wide drain, and returns non-zero after cleanup. A terminal classification gets the hard two-second abort fallback; ordinary recovery does not. The reducer re-derives the authoritative response from fresh facts on the next boot. - **`BatchSubmitter`** ([`l1/submitter/worker.rs`](../../sequencer/src/l1/submitter/worker.rs)): makes L1 progress only — never checks danger. Productive ticks re-enter immediately; idle/transient ticks sleep `idle_poll_interval`. A pure `decide_submit_start` function folds observed L1 nonces over the scheduler-accepted frontier. -- **`decide_startup_action`** ([`recovery/mod.rs`](../../sequencer/src/recovery/mod.rs)): pure function. Takes `danger` and returns `Proceed | RecoverTip { batch_index } | FlushAndCascade { batch_index } | Refuse(reason)`. L1 reachability is an execution concern: if the flush path cannot reach L1, startup fails and the orchestrator retries. +- **Startup recovery reducer** ([`recovery/mod.rs`](../../sequencer/src/recovery/mod.rs)): pure policy over one `RecoveryInspection` plus boot-local phase progress. It selects `Admit`, one phase, `Retry`, or `Refuse`. The production driver owns exhaustive error classification; raw provider/storage/flush errors do not escape to a second recovery classifier. +- **Guarded recovery storage** ([`storage/recovery.rs`](../../sequencer/src/storage/recovery.rs)): reads the reducer facts in one transaction and reasserts the selected mutation's durable preconditions in its write transaction. Divergence is checked before the flush-view coherence check and before every batch-tree mutation. - **`MempoolFlusher`** ([`recovery/flusher.rs`](../../sequencer/src/recovery/flusher.rs)): submits no-op transactions to consume all pending wallet-nonce slots and waits for safe finality. Does **not** retry internally on provider errors — the orchestrator's respawn loop is the retry mechanism. - **`ProtocolTiming`** ([`sequencer-core/src/protocol.rs`](../../sequencer-core/src/protocol.rs)): single source of truth for scheduler timing (`max_wait_blocks`) plus the sequencer-local tuning knobs (`preemptive_margin_blocks`, `l1_read_stale_after_blocks`, `seconds_per_block`). The batch-submitter address is deployment identity and is passed separately to `scheduler_accepts`. -All five pieces are replaceable at the abstraction boundary: the tick decision is a pure function; the storage surface returns structs, not ad-hoc tuples; the danger detector and submitter are independently testable. +These pieces remain independently testable: the decision is pure, the phase driver has a discriminating trace, storage returns a fact struct rather than ad-hoc tuples, and the detector/submitter remain separate workers. ## The Batch Tree @@ -47,7 +49,7 @@ The implementation handles the nonce-0 case **structurally**: `open_fresh_tip_in #### Cockroach recovery generalizes the root nonce (the anchor) -Cockroach recovery (`setup --recovery`) rebuilds a wiped DB from a trusted checkpoint and must resume submitting at nonce `N'` without replaying history — so the rebuilt tree is rooted at `N'`, not 0. Rather than plant a fake "sentinel" batch at `N'-1`, PR5 generalizes the structural root: a `batch_tree_anchor` singleton holds the nonce the parentless root carries (default `0`; recovery sets `N'`). The same `open_fresh_tip_in_tx` / `compute_next_nonce(parent = None)` path then roots `run`'s first tip at `N'`, and `trg_enforce_nonce_contiguity` validates the root against the anchor (exact match) instead of a hard-coded 0. There is **no sentinel batch row** — the root tip *is* the anchored batch. Normal deployments keep anchor `0` and are byte-identical. See [I16](../invariants.md) and the [cockroach-recovery design](#cockroach-recovery-setup---recovery) below. +Cockroach recovery (`setup --recovery`) rebuilds a wiped DB from a trusted checkpoint and must resume submitting at nonce `N'` without replaying history — so the rebuilt tree is rooted at `N'`, not 0. Rather than plant a fake "sentinel" batch at `N'-1`, the batch-tree anchor generalizes the structural root: a `batch_tree_anchor` singleton holds the nonce the parentless root carries (default `0`; recovery sets `N'`). The same `open_fresh_tip_in_tx` / `compute_next_nonce(parent = None)` path then roots `run`'s first tip at `N'`, and `trg_enforce_nonce_contiguity` validates the root against the anchor (exact match) instead of a hard-coded 0. There is **no sentinel batch row** — the root tip *is* the anchored batch. Normal deployments keep anchor `0` and are byte-identical. See [I16](../invariants.md) and the [cockroach-recovery design](#cockroach-recovery-setup---recovery) below. A sealed `N'-1` sentinel was considered and rejected: a valid closed batch at `N'-1` is a legal cascade pivot, so a runtime cascade could invalidate it and leave the tree re-rooting at 0 (ABORTed by the unchanged contiguity trigger) — an unguarded reliance on "the frontier never drops to `N'-1`". The anchor has no such hidden dependency. @@ -198,9 +200,9 @@ Stop accepting new user operations. From the outside world, the sequencer is tem ### Step 3: Flush mempool -Read the persisted **wallet-nonce watermark** `W` — the highest `w_nonce` this deployment ever broadcast (`wallet_nonce_watermark` singleton; see Implementation Constraint 1). Query the latest confirmed `w_nonce` (N) and the pending `w_nonce` (M). Submit no-op transactions (self-transfers of 0 ETH) at nonces N, N+1, ..., `max(M, W+1) - 1`. These compete with any of our transactions still alive anywhere in the network — including zombies the local node's pool has forgotten (review F1). +Read the persisted **wallet-nonce watermark** `W` — the highest `w_nonce` this deployment ever broadcast (`wallet_nonce_watermark` singleton; see Implementation Constraint 1). Query the latest confirmed `w_nonce` (N) and the pending `w_nonce` (M). Submit no-op transactions (self-transfers of 0 ETH) at nonces N, N+1, ..., `max(M, W+1) - 1`. These compete with any of our transactions still alive anywhere in the network — including zombies the local node's pool has forgotten. -Wait until both `pending <= safe` **and** `safe >= W + 1`: every slot this deployment ever used is consumed at safe depth. The second conjunct is the durable anchor — without it the flush trusts the local node's volatile mempool memory, which a dropped-locally-but-alive-elsewhere zombie evades entirely. The flush reports the safe block at which it observed resolution; Step 5 refuses to cascade until the re-synced view reaches at least that block (review F2). +Wait until both `pending <= safe` **and** `safe >= W + 1`: every slot this deployment ever used is consumed at safe depth. The second conjunct is the durable anchor — without it the flush trusts the local node's volatile mempool memory, which a dropped-locally-but-alive-elsewhere zombie evades entirely. The flush reports the safe block at which it observed resolution; Step 5 refuses to cascade until the re-synced view reaches at least that block. ### Step 4: Post-flush state @@ -235,28 +237,29 @@ A **fourth shape** sits outside this taxonomy: a closed batch that was **never s Cascading from the first non-gold catches all four. **No per-batch age check is needed for the cascade pivot itself** — every closed batch past gold is either doomed by construction or sacrificed by the convergence policy. -#### Path A — `recover_post_flush(danger_threshold)` (called from FlushAndCascade) +#### Path A — guarded post-flush Cascade After step 3 (flush) and step 4 (re-sync), the gold frontier is fresh. Run the atomic recovery transaction: 1. **Find the cascade pivot.** First try the closed pivot: first valid closed batch with `nonce >= frontier_nonce`. By the contiguity invariant, this batch's nonce is exactly `frontier_nonce`. If one exists, cascade from it. 2. **No closed pivot? Check the Tip.** When all closed batches landed fresh and were accepted (the "everything worked" aftermath), there's no closed pivot — but the Tip can still be in the danger zone. When the lane rotates without a safe-block advance between frames (e.g. immediately after init, both frames share the bootstrap `safe_block`), `S_tip = S_closed`. The closed batch can become gold by inclusion-staleness while the Tip's age — measured against `current_safe_block` after the flush wait — has crossed the danger zone. Pure monotonicity (`S_tip ≥ S_closed`) doesn't rule this out: equality is allowed. So fall through to `find_tip_batch_in_danger(danger_threshold)`. If the Tip's age clears `danger_threshold`, cascade it. -3. **Cascade-invalidate the suffix**: set `invalidated_at_ms` on every valid batch with `batch_index >= pivot.batch_index`. This catches all non-gold batches in cases (2)/(3) above, and the Tip alone in the no-pivot-but-Tip-aging case. -4. **Open recovery batch**: parent is the last valid ancestor (`MAX(batch_index) FROM valid_batches` after the cascade). Nonce is structurally `parent.nonce + 1`, which equals `frontier_nonce` — the scheduler's `expected_nonce`. Re-drain direct inputs from the invalidated batches via the `MAX(safe_input_index) + 1` query over `valid_sequenced_l2_txs`. +3. **Cascade-invalidate the suffix**: set `invalidated_at_ms` on every valid batch with `batch_index >= pivot.batch_index`. This catches all non-gold batches in cases (2)/(3) above, and the Tip alone in the no-pivot-but-Tip-aging case. The invalidation trigger retains physical replay rows but deletes their derived `executed_inputs` mappings, rewinding canonical head `H` to the surviving prefix. +4. **Advance external history reality**: iff step 3 invalidated at least one valid batch, increment `RecoveryGeneration` exactly once in this same SQLite transaction. A no-invalidation repair does not bump it. Mapping rewind and generation change are therefore one visible transition. +5. **Open recovery batch**: parent is the last valid ancestor (`MAX(batch_index) FROM valid_batches` after the cascade). Nonce is structurally `parent.nonce + 1`, which equals `frontier_nonce` — the scheduler's `expected_nonce`. Re-drain direct inputs from the invalidated batches starting at `max(base_safe_input_index, MAX(valid safe_input_index) + 1)`. Their new physical rows reuse the rewound logical offsets under the incremented generation. **Threshold = `danger_threshold`, not `MAX_WAIT_BLOCKS`**. We're already committed to recovery; the Tip is past gold; if it's also past the threshold that would have triggered recovery had it been a closed batch, cascade it. Otherwise the next danger detector tick after resume would re-trip on the Tip's eventual close + submission anyway (the closed batch would inherit its first frame's safe_block). -#### Path B — `recover_aging_tip(danger_threshold)` (called from RecoverTip) +#### Path B — guarded `RecoverTip` The `RecoverTip` action is dispatched when `check_danger` returns `TipInDanger(idx)`: no closed batch is past the gold frontier in the danger zone, but the open Tip's first frame has aged past `danger_threshold`. **No flush ran** — the Tip has no L1 footprint, so there's nothing to flush. Closed batches past gold (if any) are still in their natural lifecycle — pending in the mempool, recently included, awaiting safe finality. Cascading them would prematurely abort their progression. We act only on the Tip: -1. Run `find_tip_batch_in_danger(danger_threshold)`. If `Some(tip_index)`, cascade-invalidate from there (which only touches the Tip — no closed batches have `batch_index >= tip_index`). -2. Open a fresh recovery batch. +1. Run `find_tip_batch_in_danger(danger_threshold)`. If `Some(tip_index)`, cascade-invalidate from there (which only touches the Tip — no closed batches have `batch_index >= tip_index`) and increment `RecoveryGeneration` exactly once in that same transaction. +2. Open a fresh recovery batch in the same transaction. 3. If no Tip in danger and no Tip exists at all (torn-state crash recovery), open a Tip anyway. -The `Proceed` path does not call this function. Under that dispatch, no danger arm fired and the persisted state is left untouched; genesis Tip creation is handled by the structural `Storage::ensure_open_tip` step in `Workers::spawn` (after recovery, before the lane), not by recovery and not by the lane. +The `Safe` decision with no open Tip selects `EnsureOpenTip` as its own reducer phase. That phase rechecks `Safe`, finalized-snapshot presence, and Tip absence in the write transaction, then uses the shared `open_fresh_tip_in_tx` mechanism. Tip creation is therefore inside the same inspect → one phase → inspect discipline, never a worker-construction side effect. #### Why `danger_threshold`, not `MAX_WAIT_BLOCKS`, for the Tip threshold @@ -270,7 +273,10 @@ We invalidate at `danger_threshold` because: ### Step 6: Resume -Restart the batch submitter and user-op acceptance. The sequencer is back online. +Restart the batch submitter and user-op acceptance. If this recovery invalidated +any valid batch, the generation bump already committed atomically with that +invalidation; otherwise the history version is unchanged. The sequencer is +back online. ### Why post-flush cascade is unconditional (and not threshold-based) @@ -281,7 +287,10 @@ An earlier design considered using `MAX_WAIT_BLOCKS` as the cascade threshold ev 1. Frontier batch has `current_staleness ∈ [danger_threshold, MAX_WAIT)`. Detector trips, flush runs. 2. `recover_post_flush` (with hypothetical threshold) sees age below MAX_WAIT, declines to cascade. Resume. 3. Submitter wakes up, resubmits the Pending frontier (and any non-gold closed batches) at fresh wallet-nonce slots. They enter the mempool. -4. Detector polls again. Frontier age has barely moved (or not at all — safe head advances at ~1 block per 12s); still above `danger_threshold`. Detector trips again. +4. Detector polls again. Frontier age has barely moved or the published safe + head is unchanged; providers may later expose several newly-safe blocks as + one jump, but no cadence assumption makes the frontier clean again. It is + still above `danger_threshold`, so the detector trips again. 5. Recovery 2 starts. Flush submits no-ops at the slots the submitter just used for resubs. Bumped fees on no-ops typically out-bid resubs. Resubs killed. 6. Goto step 2. Loop converges only when `current_staleness` finally crosses `MAX_WAIT_BLOCKS` and the threshold check fires. @@ -289,44 +298,49 @@ Each loop iteration burns gas (no-ops + doomed resubs), takes ~12 minutes (the f ### Startup behavior summary -The startup flow is dispatched by `decide_startup_action(danger)`: +The first local inspection always ranks `CanonicalDivergence` and missing finalized state ahead of phase progress. If neither terminal fact exists, `NeedInitialSync` selects the initial Sync phase. A provider failure during that one phase may still admit a warm database whose persisted view remains fresh; every non-provider reader failure is classified terminal or retryable by its typed provenance. -| `check_danger` result | Action | Recovery primitive | Why this dispatch | -|---|---|---|---| -| `Safe` | `Proceed` | none | Nothing crossed danger; leave persisted state alone. The structural `ensure_open_tip` step opens the genesis Tip if the DB is fresh. | -| `L1ViewStale` | `Refuse(L1ViewStale)` | — | The L1 view is too old to support honest recovery or new soft confirmations. | -| `TipInDanger(N)` | `RecoverTip { N }` | `recover_aging_tip(danger_threshold)` (no flush — Tip has no L1 slot) | Tip has no L1 footprint; cascade and reopen directly. | -| `ClosedBatchInDanger(N)` | `FlushAndCascade { N }` | flush + `recover_post_flush(danger_threshold)` | Closed batch has L1 transactions whose fate must be resolved before cascading. | -| `EstimatedBatchInDanger(N)` | `Refuse(EstimatedBatchInDanger { N })` | — | Observed safe-state did not cross danger; only batch-relative wall-clock extrapolation did, and we don't recover from estimated state. | +After the initial Sync attempt, ordinary inspection maps facts as follows: -**L1 view freshness gates recovery.** `check_danger` first checks the L1 safe block timestamp against `l1_read_stale_after_blocks`. If the timestamp is missing or too old, startup refuses: the sequencer has no trustworthy L1 view from which to recover or issue new soft confirmations. With a fresh L1 view, observed-safe checks decide concrete recovery: `ClosedBatchInDanger` runs flush + cascade, while `TipInDanger` invalidates the open Tip directly. `EstimatedBatchInDanger` is the final batch-relative wall-clock fallback: observed safe-state has not crossed the threshold, but elapsed time since the last safe-head advance says the batch consumed its remaining runway, so startup refuses instead of recovering from estimated state. +| Local fact | Reducer decision | Why | +|---|---|---| +| `Safe` + open Tip | `Admit` | The local prediction is clean and structurally resumable. | +| `Safe` + no Tip | `EnsureOpenTip` | Open the genesis/torn-state Tip under the phase guard, then re-inspect. | +| `L1ViewStale` | `Retry` | The persisted view cannot honestly authorize new soft confirmations. | +| `TipInDanger(N)` | `RecoverTip { N }` | The Tip has no L1 footprint; invalidate and reopen directly, then re-inspect. | +| `ClosedBatchInDanger(N)` | `Flush` | Closed batches have uncertain L1 slots that must be resolved before tree mutation. | +| `EstimatedBatchInDanger(N)` | `Retry` | Observed safe state did not cross danger; recovery never mutates from an estimate alone. | +| `CanonicalDivergence(N)` | `Refuse` | Standard recovery assumes content identity and is forbidden. | + +Closed recovery is structurally `Flush → inspect → post-flush Sync → inspect → Cascade → inspect`. Flush produces a non-clone, boot-local witness carrying its observed safe block. The post-flush Sync preserves that witness, and Cascade is selected only if the persisted safe head caught up through it. A crash drops the witness, so the next boot repeats the idempotent flush instead of trusting a half-remembered phase. The guarded Cascade transaction checks divergence first, then the required finalized-state fact and the flush-view floor, then mutates. -The Refuse variants block boot and surface to the operator. `Proceed` performs no recovery writes; the genesis Tip (fresh DB) is opened by the structural `Storage::ensure_open_tip` step that runs after recovery and before the lane. The mutating recovery actions each commit atomically: `RecoverTip` invalidates the aging Tip and opens a fresh one, while `FlushAndCascade` cascades the post-flush non-gold suffix and opens a fresh Tip when needed. So the tip-existence invariant holds at every commit boundary — recovery maintains it across cascades (reopening atomically via the shared `open_fresh_tip_in_tx` mechanism), `ensure_open_tip` closes the genesis gap with the same mechanism, and the inclusion lane loads the resulting head from storage (fail-loud if absent) rather than branching on tip existence. +**Observed repair still outranks clock refusal.** `check_danger` evaluates observed closed/Tip danger before local-clock faults. Once an observed danger selected a repair, the reducer finishes that repair even if the clock arm is also active; the next mandatory inspection returns `Retry` rather than admitting. A successful repair is never itself an admission fact. -**What TLA+ proves here**: the model still abstracts away the full startup cutover/flush decision. It proves ZombieSafety once wallet-nonce slots resolve, and separately models direct recovery of an aging open Tip. The claim that past `MAX_WAIT`, closed-batch staleness self-resolves is external reasoning from L1 monotonicity. The post-flush "cascade everything past gold" choice is also external reasoning (the "everything past gold is doomed" mental model above). +After the first clean decision, runtime preparation launches zero tasks. The same reducer is invoked again after preparation, over one transactionally consistent fact set — the process lock plus the task-free prepare phase make that read the decision's linearization. Only another `Admit` decision mints the single-use `RuntimeAdmission` witness; launch consumes it synchronously. Raw component launch functions are crate-private, so external app crates can enter the runtime only through `run`/`run_main`. Runtime mutation and output authorization remains role-local at the durable boundaries in the authority ADR; the reducer establishes admission, not a new global authority service. + +The two formal models split responsibility deliberately: `preemptive.tla` proves slot/batch safety, while `admission.tla` proves local-first terminal dominance, one-phase-per-inspection ordering, witness requirements, crash/restart soundness (a crashed attempt leaves nothing behind that gates the next boot), and capability soundness. The “everything past gold is doomed” policy argument remains external to both bounded models. ### Startup observability -Startup recovery logs the decision and outcome with stable structured fields: +Startup recovery logs each reducer decision and repair outcome with stable structured fields: - `danger_status` — `safe`, `l1_view_stale`, `closed_batch_in_danger`, `tip_in_danger`, or `estimated_batch_in_danger`. - `danger_batch_index` — set for batch-specific danger statuses. -- `startup_action` — `proceed`, `recover_tip`, `flush_and_cascade`, or `refuse`. -- `refuse_reason` — present on refusal. -- `l1_reachable`, `danger_threshold`, `max_wait_blocks`, and `l1_read_stale_after_blocks` on the decision log. +- `recovery_progress` — initial sync, ordinary inspection, flushed, post-flush synced, or repaired. +- `recovery_decision` — `admit`, a single phase label, `retry`, or `refuse`. - `invalidated_count` on the completion log, plus `batches` when any batch was invalidated. -The orchestrator should still be the source of restart-loop policy and alert routing, but it should not need to parse free-form messages to distinguish "refused because the L1 view is stale" from "recovering a Tip" or "running flush + cascade." +The orchestrator remains the source of restart-loop policy and alert routing. Exit projection consumes the controller's already-classified `Retry`/`Refuse` result instead of reclassifying raw recovery errors. ### L1 view freshness -The safety policy does not branch directly on a provider-reachability boolean. Reachability is an execution concern: startup tries to sync the safe head, and if `FlushAndCascade` later cannot reach L1, the flusher errors and the orchestrator retries. The decision primitive is the freshness of the L1 view recorded in SQLite. +The safety policy does not branch directly on a provider-reachability boolean. Reachability is an execution concern: the initial Sync may fail while a warm persisted view remains usable, whereas a post-flush Sync failure must retry because Cascade requires a newly caught-up view. The decision primitive is the freshness of the L1 view recorded in SQLite plus the post-flush witness floor when one exists. The most common real-world trigger for `L1ViewStale` is a stalled RPC gateway: the provider answers, but its safe-head response stops advancing (a degraded upstream node, a load-balancer routing to a lagging replica, or a temporary indexing pause). The sequencer can't distinguish "fresh answer from a stalled view" from "L1 itself is unhealthy" without a second source of truth, so it treats both the same way: refuse to commit to soft confirmations until the recorded safe block is fresh again. -**At startup**: the sequencer attempts to sync the safe head from L1. Whether that succeeds or fails, it then checks the persisted safe block timestamp. If the timestamp is missing or older than `l1_read_stale_after_blocks * seconds_per_block`, `check_danger` returns `L1ViewStale` and startup refuses. If the view is fresh, observed-safe checks can route to recovery, and the batch-relative wall-clock estimate remains as a final refusal guard for unresolved batches whose safe-block age has effectively crossed the danger threshold. +**At startup**: the sequencer first inspects local terminal facts, then attempts the initial safe-head Sync, then inspects the persisted safe-block and progress timestamps. If the L1 timestamp is missing or older than `l1_read_stale_after_blocks * seconds_per_block`, `check_danger` returns `L1ViewStale` and startup retries. A baseline a full block-time or more ahead of `now` also yields `L1ViewStale`, but only after observed-safe checks have run. If those checks selected a repair, the repair completes and its mandatory next inspection applies the clock refusal. If the view is usable and fresh, observed-safe checks can route to recovery, and the batch-relative wall-clock estimate remains the final retry guard. -**At runtime**: the `DangerDetector` polls `Storage::check_danger` on its cadence. The input reader records both the observed safe block timestamp and the local time at which the safe head last advanced. If safe-head observations stop advancing, either the global safe block timestamp crosses the read-staleness threshold (`L1ViewStale`) or a specific unresolved batch crosses the batch-relative adjusted threshold (`EstimatedBatchInDanger`). The detector then exits with `RecoveryRequired`, the orchestrator respawns, and startup re-runs the same check. The batch submitter never observes danger; this responsibility lives entirely with the detector. +**At runtime**: the `DangerDetector` polls `Storage::check_danger` on its cadence. The input reader records both the observed safe block timestamp and the local time at which the safe head last advanced. If safe-head observations stop advancing, either the global safe block timestamp crosses the read-staleness threshold (`L1ViewStale`) or a specific unresolved batch crosses the batch-relative adjusted threshold (`EstimatedBatchInDanger`). A backward clock step of a full block-time or more against either persisted baseline also produces `L1ViewStale` — evaluated after the observed arms — and saturation must never reinterpret such a regression as zero elapsed time; sub-block steps are quantization noise for the block-granular estimate and are tolerated. The detector then exits with `RecoveryRequired`, the orchestrator respawns, and startup re-runs the same check. The batch submitter never observes danger; this responsibility lives entirely with the detector. **Other workers during L1 outages**: the inclusion lane and API are purely local (SQLite) and continue operating. The input reader retries L1 polling with error logging. All L1-dependent workers log errors at the `error` level to alert operators. @@ -347,32 +361,70 @@ Dead batches occupy `w_nonce` slots strictly below `walletNonce`. Recovery batch ## Cockroach recovery (`setup --recovery`) -Everything above is **standard recovery**: the running sequencer's own bookkeeping (the batch tree, pending dumps) lets it cascade a doomed suffix and resume. It is automatic and in-process. +Everything above is **standard recovery**: the sequencer's own bookkeeping +(the batch tree, pending dumps) lets startup cascade a doomed suffix and +resume. The repair decision is automatic, not an operator-designed reconstruction: +recovery crosses a process boundary, and the next boot inspects fresh facts +through the reducer regardless of how the prior process died: an unclean +exit leaves no gate behind, and there is no operator acknowledgement step. +A terminal death best-effort records its cause in the `terminal_faults` +black box, which nothing reads for decisions. -**Cockroach recovery** is the catastrophe path — the local DB is lost or has diverged (`CanonicalDivergence`, [I15](../invariants.md)). There is no tree to cascade; the operator wipes the DB and rebuilds canonical logical state from a trusted checkpoint plus L1. It is an operator-driven, one-shot `setup` mode, not a runtime action. The summary: +**Cockroach recovery** is the catastrophe path — the local DB is lost or has diverged (`CanonicalDivergence`, [I15](../invariants.md)). There is no tree to cascade; the operator supplies a fresh or explicitly wiped data directory and rebuilds canonical logical state from a trusted checkpoint plus L1. It is an operator-driven, one-shot `setup` mode, not a runtime action. There is no automated DB replacement, clone detection, distributed fencing, or partial-fill resume state machine. The summary: Given a trusted checkpoint machine `S` at block `B` (a finalized `dumps//` dir, carrying `N` = its resume nonce and `A` = its last-executed safe block), `setup --recovery --checkpoint-block B --checkpoint-dump-dir ` runs **flush → fold → fill**: 1. **Flush** the wallet nonce (keyed — recovery, unlike plain `setup`, signs) so every previous-instance batch resolves at safe depth `≤ C`, the post-flush safe head. Re-sync `safe_inputs` through `C`. 2. **Fold** (the pure `sequencer-core` engine, shared with the on-chain scheduler so it is consistent by construction): seed the fridge from the `(A, B]` directs (drop batches — already in `S`), replay the `(B, C]` stream, drain the leftover fridge at `C`. Yields `(S', N')` = the advanced app state and the resume nonce. -3. **Fill** a consistent DB: snapshot `S'` as finalized at `C`; **anchor the batch tree at `N'`** ([I16](../invariants.md) — the root tip *is* `N'`, no sentinel batch); sequence the `≤ C` directs so the replay cursor starts past them (they're already in `S'`, while `run`'s first on-chain batch re-drains them by `safe_block`). `run` boots from this state. +3. **Fill** a consistent DB: the baseline transaction has already minted a UUIDv4 `EraId` and initialized `RecoveryGeneration = 0`, while leaving the rebuild's `base_executed_input_count` and `base_safe_input_index` NULL. Derive `K = S'.executed_input_count()`. **Anchor the batch tree at `N'`** ([I16](../invariants.md) — the root tip *is* `N'`, no sentinel batch); sequence the `≤ C` inputs so the replay cursor starts past them (they're already in `S'`, while `run`'s first on-chain batch re-drains them by `safe_block`). Capture that root's exclusive safe-input cursor as the durable drain floor, then bind it with `K` in the same transaction that registers `S'` as the initial finalized snapshot at `C`; setup completion requires both non-NULL bases and the snapshot. Later standard recovery uses `max(base_safe_input_index, max valid attribution + 1)`, so invalidating the root cannot re-sequence those inputs. Physical `l2_tx_index` includes unmapped cursor padding and is deliberately distinct from application-history base `K`; the first executable input above the floor is mapped at `K`. `run` boots from this state. -During recovery the gold frontier (`safe_accepted_batches`) population is **deferred** (`FrontierMode::DeferUntilAnchorSet`): the tree is empty until fill, so simulating acceptance against it would flag every L1 batch as foreign and freeze the frontier ([I15](../invariants.md)). It is populated on `run`'s first sync — once the anchor `N'` is set — so the folded `< N'` history is skipped as trusted collapsed history. `N` is **trusted checkpoint metadata**, not re-verified at recovery time: a wrong-low `N` surfaces at `run` via the content-identity check, but a wrong-high `N` does not — sound because a sequencer-produced finalized dump cannot carry a wrong `N` by construction (see [`cockroach.md`](cockroach.md#data-dictionary) for the full trust boundary). Recovery is a **strict one-shot**: it refuses (terminal) on a DB that is already set up — the model is "wipe and re-run", and a crash before the `setup_complete` marker re-runs cleanly (the fill is idempotent). +During recovery the gold frontier (`safe_accepted_batches`) population is **deferred** (`FrontierMode::DeferUntilAnchorSet`): the tree is empty until fill, so simulating acceptance against it would flag every L1 batch as foreign and freeze the frontier ([I15](../invariants.md)). It is populated on `run`'s first sync — once the anchor `N'` is set — so the folded `< N'` history is skipped as trusted collapsed history. `N` is **trusted checkpoint metadata**, not re-verified at recovery time: a wrong-low `N` surfaces at `run` via the content-identity check, but a wrong-high `N` does not — sound because a sequencer-produced finalized dump cannot carry a wrong `N` by construction (see [`cockroach.md`](cockroach.md#data-dictionary) for the full trust boundary). Recovery is a **strict one-shot**: it refuses (terminal) on a DB that is already set up. A retained incomplete DB reuses the still-unexposed era minted by its baseline transaction. Once matching root Tip plus the atomically bound finalized snapshot/`K` exist, that durable fill is authoritative and retry is a no-op; it does not compare stored `K` against a later fold at a newer `C`. A fail-loud partial fill instead requires the operator to wipe and retry, minting another unexposed era. This is not general resume machinery. The detect-and-refuse gate is the *trigger*: a fresh `setup` that finds a previous instance's batches past the checkpoint refuses with exit `40` (`EXIT_SETUP_NEEDS_RECOVERY`), pointing the operator here. ## Canonical divergence (terminal, outranks every arm) Independent of the staleness machinery, the input reader's acceptance -simulation cross-checks every **accepted** landing against the local batch at -that nonce (content-identity check, review R2: `keccak256` of the landed wire -bytes vs the hash stamped at seal). On mismatch — or on an accepted landing -with no valid closed local batch at all — it persists the -`canonical_divergence` marker atomically with the sync, freezes the -acceptance frontier, and `check_danger` reports `CanonicalDivergence` -**ahead of every other arm**, so a respawn loop can never route a diverged -node into `Proceed` or `FlushAndCascade`. The startup dispatch maps it to a -terminal `Refuse`. +simulation cross-checks every at/above-anchor **accepted** landing against the +local valid closed batch at that nonce (the content-identity check: +`keccak256` of the landed wire bytes vs the hash stamped at seal). The complete +outcome set for that predicate is `Match`, `Foreign` (no local batch), or +`Mismatch` (different bytes). `Foreign`/`Mismatch` persist the +`canonical_divergence` marker in the same transaction as `safe_inputs`, the L1 +safe head, and accepted-frontier projection, and freeze the acceptance +frontier immediately. + +This automatic detection starts only when the landing reaches L1 safe and the +reader successfully ingests it. `check_danger` reports +`CanonicalDivergence` **ahead of every other arm**, so a respawn loop can never +route a diverged node into a provider call, recovery phase, or admission. Every +reducer iteration begins with local inspection; mutating phase transactions +reassert the marker's absence. The controller maps it to terminal `Refuse`. +At runtime, `DangerDetector` owns prompt process-wide reaction on its two-second +cadence. The inclusion lane's existing time-gated SQLite read independently +returns a typed divergence instead of a usable frontier, so a turn that +observes the marker closes intake and terminates before direct execution, +promotion, or the five-block frame-clock decision. This is opportunistic +refusal, not another polling schedule or reaction-time guarantee. A turn that +already read an open frontier may finish if the reader commits the marker +concurrently, and a user-op chunk committed before either runtime observation +may acknowledge; no cross-worker lock or per-chunk marker query is added. + +The operator watchdog does not replace this check. The marker freezes finalized +promotion, so the offending landing normally leaves the watchdog's checkpoint +endpoint unchanged and its idle optimization skips replay/comparison. It is a +wire-identity predicate; the watchdog is the broader independent +application-state comparison once a newer finalized checkpoint exists. + +The distinction from standard recovery is important. The danger classifier +and reducer exhaustively handle the modeled automatic-recovery states on this +page. The check is complete only for accepted-batch content identity. It is not an +independent oracle for checkpoint/application correctness, trusted collapsed +history below the anchor, the mirrored scheduler predicate, or arbitrary +direct/user execution bugs. In particular, the known wrong-high checkpoint +nonce case is outside its detection boundary (see +[`cockroach.md`](cockroach.md#data-dictionary)). Absence of the marker does not +prove general canonical agreement. The remedy is **cockroach recovery (wipe + rebuild from L1), never the standard recovery on this page**: the cascade reconciles the batch tree's @@ -385,27 +437,27 @@ local source, so rebuild-from-L1 is the only honest repair. These constraints were discovered during TLA+ model checking and are required for correctness: 1. **`walletNonce` must NOT be reset during recovery.** Recovery batches must use `w_nonces` strictly past all dead batch slots. The flush consumes dead batch slots by advancing `nextL1Slot` up to `walletNonce`. Recovery starts fresh from there. - **Mechanism (review R1a):** `walletNonce` is realized durably as the `wallet_nonce_watermark` singleton — the highest wallet nonce ever broadcast. Every broadcaster (the batch poster and the flusher's no-ops alike) commits `watermark = max(watermark, n)` power-loss-durably (`synchronous=FULL`) **before** sending at nonce `n` (write-before-broadcast; a crash between commit and send only over-covers — one wasted no-op). The flush's completion condition is `pending <= safe && safe >= watermark + 1`, so it cannot declare victory while any slot we ever used is unresolved — restoring this constraint against the local pool's volatile memory (review F1). The watermark is never reset and never lowered. + **Mechanism:** `walletNonce` is realized durably as the `wallet_nonce_watermark` singleton — the highest wallet nonce ever broadcast. Every broadcaster (the batch poster and the flusher's no-ops alike) commits `watermark = max(watermark, n)` power-loss-durably (`synchronous=FULL`) **before** sending at nonce `n` (write-before-broadcast; a crash between commit and send only over-covers — one wasted no-op). The flush's completion condition is `pending <= safe && safe >= watermark + 1`, so it cannot declare victory while any slot we ever used is unresolved — restoring this constraint against the local pool's volatile memory. The watermark is never reset and never lowered. 2. **`SubmitBatch` must use `max(walletNonce, nextL1Slot)`.** Prevents assigning `w_nonce` values for slots L1 has already consumed. 3. **`SubmitBatch` must assign ALL pending batches at once, in spine-position order.** If batches are submitted individually, a flush-win can bump one batch's `w_nonce` past a later batch's, violating the spine ordering invariant. -4. **Wall-clock freshness when the L1 view stops advancing.** The input reader records the L1 safe block timestamp and the local last-safe-head-progress time. `Storage::check_danger` first refuses on an old or missing safe block timestamp, then uses the local progress timestamp to estimate unresolved-batch age (`elapsed / seconds_per_block`). Without these checks, an L1 outage can silently push batches past the danger zone while the DB-based safe-block number remains frozen. +4. **Wall-clock freshness when the L1 view stops advancing.** The input reader records the L1 safe block timestamp and the local last-safe-head-progress time. `Storage::check_danger` first refuses on an old or missing safe block timestamp; a clock a full block-time or more out of step with either persisted baseline also refuses, but only after the observed-safe checks (sub-block skew is tolerated as quantization noise). Only a usable clock reaches the unresolved-batch estimate (`elapsed / seconds_per_block`). Without these checks, an L1 outage or a large backward clock step can silently push batches past the danger zone while the DB-based safe-block number remains frozen. 5. **The accepted-frontier cache persists acceptances, not scan progress.** `safe_accepted_batches` stores the scheduler-accepted prefix and resumes from the latest accepted safe input. Rejected batch-submitter inputs after that frontier can be rescanned on later safe-head syncs until a later batch is accepted. This is a performance tradeoff, not a correctness bug: recovery batches can reuse a scheduler nonce after earlier rejected rows, so a separate persistent scan cursor would need careful nonce-reuse tests before being introduced. ## Formal Verification -The recovery design is verified with bounded TLA+ model checking. The canonical spec is [`preemptive.tla`](preemptive.tla). An alternative optimistic design is preserved in [`history/optimistic.tla`](history/optimistic.tla). +The recovery design is verified with two complementary bounded TLA+ models. [`preemptive.tla`](preemptive.tla) owns slot/batch safety; [`admission.tla`](admission.tla) owns startup reduction and runtime admission. An alternative optimistic batch design is preserved in [`history/optimistic.tla`](history/optimistic.tla). -**Scope and limitations**: these are bounded safety models. They exhaustively check all reachable states within the configured bounds, but do not prove liveness (eventual progress), do not model the danger threshold trigger or timing margins, and do not model crash/restart (the implementation relies on SQLite atomic transactions for crash safety). +**Scope and limitations**: these are bounded safety models. They exhaustively check all reachable states within the configured bounds but do not prove liveness or model concrete timing margins. The admission model includes abstract owner loss/crash with fresh-attempt restart (there is no admission state machine to model); the slot model does not model crash/restart and relies on SQLite atomicity for its implementation mapping. ### `preemptive.tla` -- Slot-level safety under adversarial flush Models the core slot-level mechanics of preemptive recovery. At every `w_nonce` slot, L1 non-deterministically includes the spine batch OR a flush no-op (killing the batch). This covers the case where the frontier batch itself is killed during flush. The model also treats the open Tip's `safe_block` as meaningful, so it can explicitly recover an aging Tip that has no L1 footprint yet. -The model is a **safety over-approximation for the actions it shares with the implementation**: it allows `AdvanceTip` and `SubmitBatch` to interleave freely with recovery, which the real protocol prevents (the sequencer goes offline). This makes the proof stronger -- if `ZombieSafety` holds under more interleavings, it holds under fewer. However, the over-approximation claim does **not** hold action-for-action — two implementation actions sit *outside* the model's transition set: (1) the model discards an aging Tip only at `MAX_WAIT_BLOCKS`, while the implementation invalidates at `danger_threshold` (= `MAX_WAIT − MARGIN`); (2) the model's `Resolve` has no case for a killed-Pending frontier (it relies on resubmission until the frontier is Silver), while `recover_post_flush` cascades killed Pendings unconditionally. For those actions the implementation's enabled transitions are a *superset* of the model's, so TLC has not explored them; their safety rests on external arguments (an open Tip and a killed Pending have no L1/scheduler state to disagree with). The model also does not verify the full sequential protocol phases (cutover, flush, wait, recover, resume) described above; in particular, the startup decision of whether a closed unresolved batch must flush before recovery remains an external argument layered on top of the slot-level proof. +The model is a **safety over-approximation for the actions it shares with the implementation**: it allows `AdvanceTip` and `SubmitBatch` to interleave freely with recovery, which the real protocol prevents (the sequencer goes offline). This makes the proof stronger -- if `ZombieSafety` holds under more interleavings, it holds under fewer. However, the over-approximation claim does **not** hold action-for-action — two implementation actions sit *outside* the model's transition set: (1) the model discards an aging Tip only at `MAX_WAIT_BLOCKS`, while the implementation invalidates at `danger_threshold` (= `MAX_WAIT − MARGIN`); (2) the model's `Resolve` has no case for a killed-Pending frontier (it relies on resubmission until the frontier is Silver), while guarded post-flush Cascade invalidates killed Pendings unconditionally. Their safety rests on the external arguments above. Sequential startup ordering is intentionally delegated to `admission.tla` rather than cross-producting this already-large slot model. **Verified**: 157M states, 0 violations. @@ -419,10 +471,32 @@ The model is a **safety over-approximation for the actions it shares with the im | SchedulerBehindL1 | Scheduler cursor doesn't pass L1 cursor | | DeadNotYetIncluded | Dead batches have `w_nonce >= nextL1Slot` | +### `admission.tla` -- Startup reduction and authority + +Models local-first inspection, typed Retry/Refuse, InitialSync, EnsureTip, +RecoverTip, Flush/Sync/Cascade with ephemeral witnesses, Sync-discovered +divergence, mandatory reinspection after every completed phase, +crash-and-restart as a fresh attempt over surviving durable facts (nothing +durable gates the next boot; the terminal-fault black box is +write-only telemetry outside the model, and there is no acknowledgement +action), and atomic minting of the `RuntimeAdmission` witness from the +final clean decision. It abstracts away the batch spine and delegates every phase's +batch mechanics to `preemptive.tla`. + +**Verified**: 860 generated states, 266 distinct states, depth 13, 0 violations. + +Key invariants include capability soundness, reducer/phase control while an +attempt is begun, completed-phase reinspection, terminal dominance, Cascade +witness preconditions, and Retry never being interpreted as clean. Concrete SQLite +mutation guards and transaction atomicity are tested in Rust rather than +modeled as database actions here. + ### Running the spec ```bash +tlc -workers auto -deadlock docs/recovery/admission.tla tlc -workers auto -deadlock docs/recovery/preemptive.tla # ~90s +just -f docs/recovery/justfile check-all ``` -Bounds are in `preemptive.cfg`. The `MaxWalletNonce` bound keeps the state space finite (kill/resubmit cycles generate new `w_nonce` values). Increase bounds for higher confidence at the cost of longer runtime. +Bounds are in `admission.cfg` and `preemptive.cfg`. The `MaxWalletNonce` bound keeps the slot model finite (kill/resubmit cycles generate new `w_nonce` values). Increase bounds for higher confidence at the cost of longer runtime. diff --git a/docs/recovery/admission.cfg b/docs/recovery/admission.cfg new file mode 100644 index 00000000..08218d7f --- /dev/null +++ b/docs/recovery/admission.cfg @@ -0,0 +1,4 @@ +SPECIFICATION Spec + +INVARIANTS + Inv diff --git a/docs/recovery/admission.tla b/docs/recovery/admission.tla new file mode 100644 index 00000000..a2a89cfd --- /dev/null +++ b/docs/recovery/admission.tla @@ -0,0 +1,522 @@ +----------------------------- MODULE admission ----------------------------- +(* + * Run-specific admission model for the startup recovery controller. + * + * Batch/L1 slot mechanics stay in preemptive.tla. This model checks the + * controller protocol implemented by recovery/mod.rs and commands/run/: + * + * inspect one local SQLite fact set -> classify once -> decide + * -> perform at most one phase -> inspect again + * + * The first clean decision grants no authority. It starts fallible, task-free + * Prepare, whose successful completion also returns to local inspection. Only + * a second clean decision mints the single-use RuntimeAdmission witness. + * + * Admission gating is fact-derived: there is no + * lifecycle admission state machine and no acknowledgement step, and no + * durable per-attempt record gates anything (the + * terminal-fault black box is write-only telemetry outside this model). + * Settlement and crash both end the attempt, and the next boot begins fresh + * over whatever facts persist. Divergence and danger facts are durable and + * survive attempts; everything else is boot-local. + * + * Flushed and PostFlushSynced abstract the non-clone session witnesses carried + * by the Rust controller. They are ephemeral: crash, Retry, and Refuse erase + * them, so another attempt must flush again before it can cascade. + *) + +EXTENDS TLC + +Idle == "Idle" +InspectLocal == "InspectLocal" +Decide == "Decide" +Prepare == "Prepare" +InitialSync == "InitialSync" +EnsureOpenTip == "EnsureOpenTip" +RecoverTip == "RecoverTip" +Flush == "Flush" +PostFlushSync == "PostFlushSync" +Cascade == "Cascade" +Admitted == "Admitted" + +PhasePC == {InitialSync, EnsureOpenTip, RecoverTip, Flush, PostFlushSync, + Cascade} +StartupPC == {InspectLocal, Decide, Prepare} \union PhasePC +ControllerStates == {Idle, Admitted} \union StartupPC + +NoProgress == "NoProgress" +NeedInitialSync == "NeedInitialSync" +Inspecting == "Inspecting" +Flushed == "Flushed" +PostFlushSynced == "PostFlushSynced" +Repaired == "Repaired" + +ProgressStates == {NoProgress, NeedInitialSync, Inspecting, Flushed, + PostFlushSynced, Repaired} + +Safe == "Safe" +ClosedDanger == "ClosedDanger" +TipDanger == "TipDanger" +RetryDanger == "RetryDanger" + +DangerStates == {Safe, ClosedDanger, TipDanger, RetryDanger} + +NoPostFlushView == "NoPostFlushView" +CaughtUp == "CaughtUp" +Behind == "Behind" +MissingSafeHead == "MissingSafeHead" + +PostFlushViews == {CaughtUp, Behind, MissingSafeHead} + +NoDecision == "NoDecision" +Admit == "Admit" +Retry == "Retry" +Refuse == "Refuse" + +Decisions == {NoDecision, Admit, Retry, Refuse} \union PhasePC + +VARIABLES + controller, + admittedRuntime, + prepared, + progress, + danger, + hasFinalizedSnapshot, + hasOpenTip, + postFlushView, + canonicalDivergence, + decision, + mustInspect + +vars == <> + +HasFlushWitness == progress \in {Flushed, PostFlushSynced} +HasPostFlushSyncWitness == progress = PostFlushSynced + +Reduce(currentProgress, currentDanger, snapshotPresent, tipPresent, + currentPostFlushView, diverged) == + IF diverged \/ ~snapshotPresent + THEN Refuse + ELSE CASE currentProgress = NeedInitialSync -> InitialSync + [] currentProgress = Flushed -> PostFlushSync + [] currentProgress = PostFlushSynced -> + CASE currentPostFlushView = MissingSafeHead -> Refuse + [] currentPostFlushView = Behind -> Retry + [] OTHER -> Cascade + [] currentProgress = Repaired -> + CASE currentDanger = Safe /\ tipPresent -> Admit + [] currentDanger = Safe -> EnsureOpenTip + [] OTHER -> Retry + [] currentProgress = Inspecting -> + CASE currentDanger = Safe /\ tipPresent -> Admit + [] currentDanger = Safe -> EnsureOpenTip + [] currentDanger = ClosedDanger -> Flush + [] currentDanger = TipDanger -> RecoverTip + [] OTHER -> Retry + [] OTHER -> Refuse + +--------------------------------------------------------------------------- +(* Initial state: a boot begins over any persisted fact shape, including + * pre-existing divergence. TLC explores all locally inspectable fact + * shapes. *) + +Init == + /\ controller = Idle + /\ admittedRuntime = FALSE + /\ prepared = FALSE + /\ progress = NoProgress + /\ danger \in DangerStates + /\ hasFinalizedSnapshot \in BOOLEAN + /\ hasOpenTip \in BOOLEAN + /\ postFlushView = NoPostFlushView + /\ canonicalDivergence \in BOOLEAN + /\ decision = NoDecision + /\ mustInspect = FALSE + +(* Retry/Refuse ends the attempt. Prepared resources and session witnesses do + * not cross that boundary; the next attempt begins fresh. *) +Settle == + /\ controller' = Idle + /\ admittedRuntime' = FALSE + /\ prepared' = FALSE + /\ progress' = NoProgress + /\ postFlushView' = NoPostFlushView + /\ decision' = NoDecision + /\ mustInspect' = FALSE + /\ UNCHANGED <> + +--------------------------------------------------------------------------- +(* Attempt begin and the single local inspection step. Begin has no + * lifecycle-state precondition: the fact gates the code checks here — + * two-sided setup completion — are outside this model's scope, and the + * kernel process lock excludes a concurrent owner. *) + +BeginRun == + /\ controller = Idle + /\ controller' = InspectLocal + /\ admittedRuntime' = FALSE + /\ prepared' = FALSE + /\ progress' = NeedInitialSync + /\ postFlushView' = NoPostFlushView + /\ decision' = NoDecision + /\ mustInspect' = TRUE + /\ UNCHANGED <> + +(* One SQLite RecoveryInspection is the only input to Reduce. Persisted + * divergence and missing finalized state are classified by the same call. *) +InspectFacts == + /\ controller = InspectLocal + /\ controller' = Decide + /\ decision' = Reduce(progress, danger, hasFinalizedSnapshot, + hasOpenTip, postFlushView, + canonicalDivergence) + /\ mustInspect' = FALSE + /\ UNCHANGED <> + +(* Storage-open/query failures are centrally classified. A known local + * divergence cannot be masked by the retry edge. *) +InspectRetry == + /\ controller = InspectLocal + /\ ~canonicalDivergence + /\ Settle + +InspectRefuse == + /\ controller = InspectLocal + /\ Settle + +--------------------------------------------------------------------------- +(* Decision handling before preparation. *) + +DecidePhase == + /\ controller = Decide + /\ ~prepared + /\ decision \in PhasePC + /\ controller' = decision + /\ UNCHANGED <> + +DecideRetry == + /\ controller = Decide + /\ decision = Retry + /\ Settle + +DecideRefuse == + /\ controller = Decide + /\ decision = Refuse + /\ Settle + +(* The first clean decision begins authority-neutral, task-free preparation. *) +BeginPrepare == + /\ controller = Decide + /\ ~prepared + /\ decision = Admit + /\ controller' = Prepare + /\ UNCHANGED <> + +(* If the final inspection no longer says Admit, prepared resources are + * dropped and the attempt exits; no new recovery phase runs on the aged + * prepared state. *) +PreparedDecisionChanged == + /\ controller = Decide + /\ prepared + /\ decision \in PhasePC + /\ Settle + +(* The capability boundary: the final clean decision and the RuntimeAdmission + * witness are one atomic action; launch consumes the witness without + * yielding. *) +AdmitRuntime == + /\ controller = Decide + /\ prepared + /\ decision = Admit + /\ controller' = Admitted + /\ admittedRuntime' = TRUE + /\ UNCHANGED <> + +--------------------------------------------------------------------------- +(* Recovery phase completion. Every successful phase returns to InspectLocal. + * InitialSync and PostFlushSync may update observed danger facts; either Sync + * may also discover canonical divergence. *) + +CompletePhase(nextProgress, nextDanger, nextTipPresent, nextPostFlushView, + discoversDivergence) == + /\ controller \in PhasePC + /\ ~prepared + /\ ~canonicalDivergence + /\ controller' = InspectLocal + /\ progress' = nextProgress + /\ danger' = nextDanger + /\ hasOpenTip' = nextTipPresent + /\ postFlushView' = nextPostFlushView + /\ canonicalDivergence' = discoversDivergence + /\ decision' = NoDecision + /\ mustInspect' = TRUE + /\ UNCHANGED <> + +InitialSyncCompleted == + /\ controller = InitialSync + /\ \E nextDanger \in DangerStates: + CompletePhase(Inspecting, nextDanger, hasOpenTip, + NoPostFlushView, FALSE) + +EnsureOpenTipCompleted == + /\ controller = EnsureOpenTip + /\ CompletePhase(Repaired, Safe, TRUE, NoPostFlushView, FALSE) + +(* The two repair completions restrict post-repair facts deliberately: these + * are faithful storage postconditions, not narrowing. `hasOpenTip' = TRUE` + * because `recover_aging_tip_for_recovery` / `cascade_and_reopen` end with a + * valid open batch in the same transaction (storage/recovery.rs). Danger is + * `{Safe, RetryDanger}`: RecoverTip fires only after the closed frontier was + * checked clean and its cascade touches only `>= tip`, Cascade invalidates + * the whole non-gold closed suffix, the fresh tip's first frame carries the + * current safe block, and no repair phase contacts L1 — so ClosedDanger / + * TipDanger cannot reappear and only the retryable observations remain. + * Widening either action would model states the implementation cannot + * produce. *) +RecoverTipCompleted == + /\ controller = RecoverTip + /\ \E nextDanger \in {Safe, RetryDanger}: + CompletePhase(Repaired, nextDanger, TRUE, + NoPostFlushView, FALSE) + +FlushCompleted == + /\ controller = Flush + /\ CompletePhase(Flushed, danger, hasOpenTip, + NoPostFlushView, FALSE) + +PostFlushSyncCompleted == + /\ controller = PostFlushSync + /\ \E nextDanger \in DangerStates: + \E nextView \in PostFlushViews: + CompletePhase(PostFlushSynced, nextDanger, hasOpenTip, + nextView, FALSE) + +CascadeCompleted == + /\ controller = Cascade + /\ \E nextDanger \in {Safe, RetryDanger}: + CompletePhase(Repaired, nextDanger, TRUE, + NoPostFlushView, FALSE) + +SyncDiscoversDivergence == + \/ /\ controller = InitialSync + /\ \E nextDanger \in DangerStates: + CompletePhase(Inspecting, nextDanger, hasOpenTip, + NoPostFlushView, TRUE) + \/ /\ controller = PostFlushSync + /\ \E nextDanger \in DangerStates: + \E nextView \in PostFlushViews: + CompletePhase(PostFlushSynced, nextDanger, hasOpenTip, + nextView, TRUE) + +PhaseRetry == + /\ controller \in PhasePC + /\ ~canonicalDivergence + /\ Settle + +PhaseRefuse == + /\ controller \in PhasePC + /\ ~canonicalDivergence + /\ Settle + +--------------------------------------------------------------------------- +(* Fallible task-free preparation. Time may pass, so the next inspection may + * derive a different danger status even though preparation changes no local + * recovery fact itself. *) + +PrepareCompleted == + /\ controller = Prepare + /\ ~prepared + /\ decision = Admit + /\ \E nextDanger \in DangerStates: + /\ controller' = InspectLocal + /\ prepared' = TRUE + /\ progress' = Inspecting + /\ danger' = nextDanger + /\ decision' = NoDecision + /\ mustInspect' = TRUE + /\ UNCHANGED <> + +PrepareRetry == + /\ controller = Prepare + /\ ~canonicalDivergence + /\ Settle + +PrepareRefuse == + /\ controller = Prepare + /\ ~canonicalDivergence + /\ Settle + +--------------------------------------------------------------------------- +(* Crash destroys PreparedRuntime, the RuntimeAdmission witness, and session + * witnesses. + * Nothing durable gates the next boot (the terminal-fault black box + * is write-only telemetry), so a restart is simply a fresh attempt over the + * surviving durable facts. Modeled as returning directly to the pre-begin + * shape with facts unchanged. *) + +Crash == + /\ controller \in StartupPC \union {Admitted} + /\ Settle + +CleanShutdown == + /\ controller = Admitted + /\ admittedRuntime + /\ Settle + +--------------------------------------------------------------------------- +(* Safety invariants. *) + +TypeOK == + /\ controller \in ControllerStates + /\ admittedRuntime \in BOOLEAN + /\ prepared \in BOOLEAN + /\ progress \in ProgressStates + /\ danger \in DangerStates + /\ hasFinalizedSnapshot \in BOOLEAN + /\ hasOpenTip \in BOOLEAN + /\ postFlushView \in PostFlushViews \union {NoPostFlushView} + /\ canonicalDivergence \in BOOLEAN + /\ decision \in Decisions + /\ mustInspect \in BOOLEAN + +ControllerShape == + /\ controller = Idle => + /\ ~admittedRuntime + /\ ~prepared + /\ progress = NoProgress + /\ controller \in StartupPC => ~admittedRuntime + +ProgressShape == + /\ controller \in StartupPC => progress \in ProgressStates \ {NoProgress} + /\ controller = Admitted => progress = Inspecting + /\ postFlushView # NoPostFlushView <=> HasPostFlushSyncWitness + /\ prepared => + /\ progress = Inspecting + /\ controller \in {InspectLocal, Decide, Admitted} + +AdmittedRuntimeSound == + admittedRuntime => + /\ controller = Admitted + /\ prepared + /\ progress = Inspecting + /\ decision = Admit + /\ decision = Reduce(progress, danger, hasFinalizedSnapshot, + hasOpenTip, postFlushView, + canonicalDivergence) + /\ danger = Safe + /\ hasFinalizedSnapshot + /\ hasOpenTip + /\ ~canonicalDivergence + +AdmissionIsAtomic == + controller = Admitted => admittedRuntime + +MandatoryReinspection == + mustInspect => + /\ controller = InspectLocal + /\ ~admittedRuntime + +ClassifiedOnce == + controller = Decide => + decision = Reduce(progress, danger, hasFinalizedSnapshot, + hasOpenTip, postFlushView, + canonicalDivergence) + +EphemeralWitnessScope == + /\ HasPostFlushSyncWitness => HasFlushWitness + /\ HasFlushWitness => + /\ ~prepared + /\ controller \in StartupPC + /\ controller = PostFlushSync => progress = Flushed + /\ controller = Cascade => + /\ progress = PostFlushSynced + /\ postFlushView = CaughtUp + +LocalDivergenceFirst == + /\ canonicalDivergence => ~admittedRuntime + /\ canonicalDivergence /\ controller = Decide => decision = Refuse + /\ controller \in ({Prepare, Admitted} \union PhasePC) => + ~canonicalDivergence + +PhasePreconditions == + /\ controller \in PhasePC => decision = controller + /\ controller = InitialSync => progress = NeedInitialSync + /\ controller = EnsureOpenTip => + /\ progress \in {Inspecting, Repaired} + /\ danger = Safe + /\ ~hasOpenTip + /\ controller = RecoverTip => + /\ progress = Inspecting + /\ danger = TipDanger + /\ controller = Flush => + /\ progress = Inspecting + /\ danger = ClosedDanger + +PrepareRequiresFirstCleanInspection == + controller = Prepare => + /\ ~prepared + /\ progress \in {Inspecting, Repaired} + /\ decision = Admit + /\ danger = Safe + /\ hasFinalizedSnapshot + /\ hasOpenTip + /\ ~canonicalDivergence + +Inv == + /\ TypeOK + /\ ControllerShape + /\ ProgressShape + /\ AdmittedRuntimeSound + /\ AdmissionIsAtomic + /\ MandatoryReinspection + /\ ClassifiedOnce + /\ EphemeralWitnessScope + /\ LocalDivergenceFirst + /\ PhasePreconditions + /\ PrepareRequiresFirstCleanInspection + +--------------------------------------------------------------------------- + +Next == + \/ BeginRun + \/ InspectFacts + \/ InspectRetry + \/ InspectRefuse + \/ DecidePhase + \/ DecideRetry + \/ DecideRefuse + \/ BeginPrepare + \/ PreparedDecisionChanged + \/ AdmitRuntime + \/ InitialSyncCompleted + \/ EnsureOpenTipCompleted + \/ RecoverTipCompleted + \/ FlushCompleted + \/ PostFlushSyncCompleted + \/ CascadeCompleted + \/ SyncDiscoversDivergence + \/ PhaseRetry + \/ PhaseRefuse + \/ PrepareCompleted + \/ PrepareRetry + \/ PrepareRefuse + \/ Crash + \/ CleanShutdown + +Spec == Init /\ [][Next]_vars + +============================================================================= diff --git a/docs/recovery/cockroach.md b/docs/recovery/cockroach.md index 6bc6b6f6..e4f54550 100644 --- a/docs/recovery/cockroach.md +++ b/docs/recovery/cockroach.md @@ -4,12 +4,16 @@ The catastrophe path. When the local DB is lost or has diverged ([`CanonicalDivergence`](../invariants.md#i15-divergence-marker-present--acceptance-frontier-frozen)), there is no batch tree to cascade — the operator **wipes the data dir and rebuilds canonical logical state from a trusted checkpoint plus L1**. It is an -operator-driven, one-shot `setup` mode, not a runtime action. +operator-driven, one-shot `setup` mode, not a runtime action. There is no +automated database replacement, clone detector, distributed fence, or +partial-fill resume state machine: the supported repair is deliberately the +explicit fresh/wiped-directory flow. Contrast with **[standard / preemptive recovery](README.md)** (the rest of `docs/recovery/`): that runs *inside* a live sequencer, uses its own batch tree to cascade a doomed suffix, and shares the flush machinery. Cockroach recovery -discards the tree entirely and reconstructs `(S', N')` by *folding* L1. +discards the tree entirely and reconstructs `(S', N')` by *folding* L1, then +records the recovered application's absolute executed-input base `K`. The pure fold engine ([`sequencer-core/src/scheduler/fold.rs`](../../sequencer-core/src/scheduler/fold.rs)) is the same scheduler source compiled into the on-chain canonical machine — so @@ -20,8 +24,8 @@ re-implementation. ## Data dictionary -Five quantities drive the procedure. Knowing where each is *born* is the key to -reading the code. +The fold/fill quantities and history metadata below drive the procedure. +Knowing where each is *born* is the key to reading the code. | Symbol | Meaning | Where it comes from | |---|---|---| @@ -29,8 +33,11 @@ reading the code. | **`A`** | `S`'s last-executed safe block. The fridge is reconstructed from directs in `(A, B]`. | App query: `S.last_executed_safe_block()`. (Persisted in the dump — see `docs/snapshots/format.md`.) | | **`B`** | The checkpoint's L1 inclusion block. `S` reflects every **batch** with inclusion `≤ B` and **no direct** in `(A, B]`. | Operator arg: `--checkpoint-block`. | | **`N`** | The checkpoint's resume batch nonce (the scheduler counter at `B`). The bare-metal app *cannot* recompute it, so it rides as checkpoint metadata. | `info.toml`'s `next_batch_nonce` in the dump. | -| **`C`** | The post-flush safe head — the stopping block. The flush guarantees every previous-instance batch is settled at safe depth `≤ C`. | Return of `flusher.flush_and_wait(...)`. | +| **`C`** | The post-flush safe head — the stopping block. The flush resolves every slot the provider remembers at safe depth `≤ C` (best-effort — see step 2). | Return of `flusher.flush_and_wait(...)`. | | **`N'`** | The resume nonce the new sequencer submits at — `N` advanced by the accepted batches in `(B, C]`. Becomes the **batch-tree anchor**. | Output of `fold_replay(...)`. | +| **`E`, `g`** | The new history era and its recovery generation. `E` is UUIDv4; `g = 0`. | Minted with the baseline schema in one transaction, before external recovery work. | +| **`K`** | The first application-history offset available in `E`: `S'.executed_input_count()`. It is not the replacement DB's physical replay cursor. | Derived after the fold and bound atomically with the initial finalized snapshot row. | +| **`F`** | The exclusive `safe_inputs` cursor already represented by `S'`; standard recovery must never drain below it. | Captured after the recovery root sequences its `≤ C` cursor padding and bound atomically with `K` and the initial finalized snapshot row. | **Invariant `A < B`** (checked at load): the checkpoint's executed state must predate its inclusion block, or the `(A, B]` fridge range is ill-defined. @@ -60,10 +67,21 @@ symmetric: against itself while the real scheduler ignores it. This is why the checkpoint must be a trustworthy finalized dump, not merely "some app bytes". +This is a concrete boundary of the content-identity check: it is complete for at/above-anchor accepted +batch content identity, not checkpoint/application correctness or arbitrary +canonical divergence. Absence of `canonical_divergence` is not a proof that a +checkpoint outside the trust boundary was valid. + --- ## The procedure: flush → fold → fill +Opening the fresh replacement DB first commits one baseline transaction: the +schema and a UUIDv4 era with generation zero. Because neither the folded +application nor the recovery-root +cursor exists yet, `base_executed_input_count` and `base_safe_input_index` start +NULL. That era remains externally unexposed until setup completes. + ``` ┌─ load S, derive A & N, require A < B checkpoint │ @@ -87,10 +105,23 @@ symmetric: 1. **Load `S`; derive `A`, `N`; require `A < B`.** Read the dump (`from_dump` + `info.toml`); `A = S.last_executed_safe_block()`. 2. **Flush → `C`.** Settle the wallet nonce (keyed L1 no-ops; this is where - cockroach recovery composes with the standard flush). On completion every - previous-instance batch is resolved at safe depth `≤ C`. `C` is the stopping + cockroach recovery composes with the standard flush). `C` is the stopping point: directs beyond `C` are `run`'s job, not the fold's. -3. **Re-sync `safe_inputs`; F2 coherence.** The reader syncs to the *live* safe + **This flush is best-effort by construction:** the wiped DB carries no + wallet-nonce watermark, so the durable-anchor half of the completion test + is vacuous — the flush resolves only the slots the provider remembers, + and a zombie tx the local node forgot but the network still holds is + unresolvable here (plain `setup`'s detection gate shares the same false + negative). The content-identity check is what makes this acceptable: such + a zombie landing at/above `N'` is detected and freezes the frontier + instead of silently diverging, and the repair is another wipe-and-rerun — + cockroach recovery recovers from the failure of its own flush. If ever + needed, an operator-supplied flush floor taken from the old DB's + watermark is a sound option: the value is fail-safe under corruption + (too high wastes a few no-ops; too low degrades to exactly best-effort), + so reading it from an untrusted half-destroyed DB does not violate the + don't-trust-local-state premise. +3. **Re-sync `safe_inputs`; flush-view coherence.** The reader syncs to the *live* safe head `H1` (normally `> C` — real time passed while the flush awaited safe finality); refuse only if it *lags* `C` (a load-balanced RPC replica could serve a stale view). So `safe_inputs` ends up holding directs through `H1`, @@ -106,20 +137,27 @@ symmetric: replays `(B, C]` (force-executing overdue directs, applying accepted batches, draining covered fridge directs), drains the leftover fridge at `C`, and advances the nonce to `N'`. -6. **Fill the DB.** Snapshot `S'` as finalized at `C`; **anchor the batch tree +6. **Fill the DB.** Derive `K = S'.executed_input_count()`. **Anchor the batch tree at `N'`** (the root tip *is* `N'` — there is no sentinel batch); open the root tip at frame `safe_block = C` and sequence **only the `≤ C` safe inputs** so the replay cursor starts past them. The drain is sender-unfiltered — it includes the `≤ C` batch-submitter rows alongside the user directs the fold folded into `S'`, exactly as the genesis tip drains its whole span; those rows are sequenced (cursor padding), never executed, so the - `sender != batch_submitter` *seed* filter does not reappear here. The `(C, H1]` + `sender != batch_submitter` *seed* filter does not reappear here. Capture the + root's exclusive safe-input cursor as `F`, then snapshot `S'` as finalized at + `C` and bind `(K, F)` in the same SQLite transaction; setup completion + refuses until both bases and the finalized snapshot exist. The `(C, H1]` directs the resync pulled in past `C` stay **undrained** — `run`'s lane leads and executes them exactly once as the safe frontier advances `C → H1`. (Draining them here instead would skip them on catch-up while `S'` never executed them — a vanished deposit / divergence; this is why the fill uses a `≤ C`-capped `open_recovery_tip`, not the generic whole-table drain.) `run` - boots from this state. + boots from this state. Those padding rows advance physical `l2_tx_index`, + but not `K` and receive no `executed_inputs` mapping: they are physical + cursor attribution for inputs already reflected in `S'`, not newly executed + application history. The first executable direct above `F` receives logical + offset `K`; applying it moves the recovered application to `K + 1`. --- @@ -127,13 +165,14 @@ symmetric: | Step | Code | |---|---| -| entry / branch | [`setup.rs` `setup()`](../../sequencer/src/runtime/setup.rs) branches on `config.recovery` after the shared prefix (identity pin + initial sync) | -| 1. load + `A < B` | [`recover()` step 1](../../sequencer/src/runtime/setup.rs) — `from_dump`, `read_info`, `CheckpointNotBeforeBlock` | -| 2. flush → `C` | `recover()` step 2 — `MempoolFlusher::flush_and_wait` (see [`runtime/flush.rs`](../../sequencer/src/runtime/flush.rs)) | +| entry / branch | [`setup()`](../../sequencer/src/commands/setup/mod.rs) branches on `config.recovery` after the shared prefix (identity pin + initial sync) | +| 1. load + `A < B` | [`recover()` step 1](../../sequencer/src/commands/setup/mod.rs) — `from_dump`, `read_info`, `CheckpointNotBeforeBlock` | +| 2. flush → `C` | `recover()` step 2 — `MempoolFlusher::flush_and_wait` (see [`recovery/flusher.rs`](../../sequencer/src/recovery/flusher.rs)) | | 3. re-sync + coherence | `recover()` step 3 — `set_frontier_mode(DeferUntilAnchorSet)` + `sync_to_current_safe_head` + `ResyncBehindFlushView` | | 4. source seeds/replay | `recover()` step 4 — `Storage::safe_inputs_in_block_range` + the `sender != submitter` filter + `to_fold_input` | | 5. fold | [`fold_replay`](../../sequencer-core/src/scheduler/fold.rs) | -| 6. fill | [`fill_recovery_state`](../../sequencer/src/runtime/setup_fill.rs) — anchor + `open_recovery_tip` (`≤ C`-capped drain at frame `safe_block = C`) + finalized snapshot | +| baseline history | [`baseline_migration`](../../sequencer/src/storage/open.rs) — UUIDv4 era + generation zero + NULL rebuild base in one baseline transaction | +| 6. fill | [`fill_recovery_state`](../../sequencer/src/commands/setup/fill.rs) — anchor + `open_recovery_tip` (`≤ C`-capped drain at frame `safe_block = C`) + atomic finalized-snapshot/`K` bind | | anchor mechanism | [`trg_enforce_nonce_contiguity`](../../sequencer/src/storage/migrations/0001_schema.sql) + `compute_next_nonce` + the anchor-aware frontier in [`safe_accepted_batches.rs`](../../sequencer/src/storage/safe_accepted_batches.rs) | --- @@ -157,8 +196,19 @@ symmetric: sequencing, so `run`'s catch-up (`offset > l2_tx_index`) skips them; they are already in `S'`. Symmetrically, the `(C, H1]` directs are **not** sequenced here (`open_recovery_tip` caps the drain at `C`), so the cursor sits below them - and `run`'s lane leads + executes them exactly once — neither double-executed - nor lost. + and `run`'s lane leads + executes them exactly once — neither double-executed + nor lost. The durable `base_safe_input_index = F` also survives invalidation + of the recovery root: subsequent standard recovery derives its drain cursor + as `max(F, max valid attribution + 1)` and cannot re-sequence or re-execute + the `≤ C` prefix after the root's padding leaves the valid view. +- **History base is application state, not cursor padding** — rebuild baseline + leaves `(K, F)` NULL; fill derives `K` from `S'.executed_input_count()` and + `F` from the root's exclusive safe-input cursor, then binds both with the + initial finalized snapshot. Setup requires all three before completion. Physical + `l2_tx_index` remains a rowid replay cursor and may be greater because it also + covers sequenced-but-not-executed batch-envelope padding. Those padding rows + are deliberately absent from the canonical mapping; new executable history + is attributed contiguously from `K`. --- @@ -168,10 +218,16 @@ symmetric: - It **refuses** (terminal, exit 30) if `setup_complete` already exists — the model is "delete the data dir and re-run", not resume-a-live-deployment. -- The `setup_complete` marker is the linearization point, written **last**. +- The `setup_complete` marker is the linearization point, written **last**; it + requires non-NULL `K`, non-NULL `F`, and the finalized snapshot. - A crash *before* the marker is handled fail-loud, not by blind resume: + - Retaining an incomplete DB retains the UUIDv4 era minted by its baseline + transaction. An early retry therefore reuses that still-unexposed era; it + does not rotate merely because the command restarted. - A **completed** fill (finalized snapshot present — the last write) re-runs as - a safe no-op; the anchor and root tip are already in place. + a safe no-op once the root Tip's `N'` matches. Its atomically bound + finalized snapshot/`(K, F)` tuple is authoritative; retry never compares + that stored base with a later fold at a newer `C`. - A **same-`N'`** re-run of a fill that crashed *mid-fill* (root tip exists, no finalized snapshot) is **refused** (`PartialRecoveryIncomplete`). It is *not* idempotent: a re-sync may have advanced `C` with new directs (which leave @@ -191,6 +247,11 @@ symmetric: (`GenesisOverRecoveryResidue`, anchor `≠ 0`) — it must not root genesis at the recovery nonce. +Any fail-loud partial-fill refusal requires the explicit operator wipe/retry. +That fresh baseline necessarily mints another era, while the discarded era was +never exposed by a completed setup. This narrow completed-fill no-op is not a +general resumable rebuild protocol. + (See the deep-review remediations: these guards close the partial-recovery revalidation gap, including the same-`N'`/advanced-`C` double-drain — external review 2026-06.) diff --git a/docs/recovery/justfile b/docs/recovery/justfile index a35604ad..a069aa0e 100644 --- a/docs/recovery/justfile +++ b/docs/recovery/justfile @@ -1,5 +1,9 @@ tlc := env("TLC", "tlc") +# Check startup reducer and runtime admission safety +check-admission: + {{tlc}} -workers auto -deadlock admission.tla + # Check the preemptive recovery spec (~90s) check-preemptive: {{tlc}} -workers auto -deadlock preemptive.tla @@ -9,4 +13,4 @@ check-optimistic: {{tlc}} -workers auto -deadlock history/optimistic.tla # Check all specs -check-all: check-preemptive check-optimistic +check-all: check-admission check-preemptive check-optimistic diff --git a/docs/review/2026-06-10-correctness-review.md b/docs/review/2026-06-10-correctness-review.md new file mode 100644 index 00000000..4c22c1f8 --- /dev/null +++ b/docs/review/2026-06-10-correctness-review.md @@ -0,0 +1,19 @@ +# Whole-project correctness review (2026-06-10) + +Twelve parallel module reviews plus line-by-line passes over the core files, +every medium/high concern independently adversarially verified. **Headline:** +the sequencer/scheduler duality itself was in good shape; the confirmed +problems clustered at the boundary with the infrastructure underneath — +fsync semantics, the local node's mempool memory, RPC fleet coherence, and +the subscriber protocol. + +Ten findings (F1–F10) and five design resolutions (R1–R5) came out of it; +all findings except the WS invalidation contract (owned by the Track 3 +handoff) are fixed, and the resolutions became the write-before-broadcast +watermark (I14), the content-identity check (I9/I15), `synchronous=FULL`, +the exit-code contract, and the fail-loud check policy that now opens +[`docs/invariants.md`](../invariants.md). + +Everything still actionable — the remaining robustness/hygiene backlog and +the refuted concerns that must not be re-litigated — lives in +[`register.md`](register.md); the full original ledger is in git history. diff --git a/docs/review/2026-06-10-simplification.md b/docs/review/2026-06-10-simplification.md new file mode 100644 index 00000000..2f65cfa0 --- /dev/null +++ b/docs/review/2026-06-10-simplification.md @@ -0,0 +1,18 @@ +# Simplification & refactoring review (2026-06-10) + +Companion to the correctness review: what was *heavier than it needed to +be*, ranked and sequenced. **Verdict: no architectural restructure** — the +module layout (one file per writer role, `*_in(tx)` free functions composing +into larger transactions, storage-owns-SQLite / lane-owns-filesystem) is +sound and should be defended, not redesigned. The weight traced to unpinned +cross-file invariants (answered by creating +[`docs/invariants.md`](../invariants.md)), test-only surface presenting as +production API, and duplicated semantics — each with a single-home fix. + +The queue's durable outcomes: the scheduler-mirroring logic homed beside +`scheduler_accepts`, the shared recovery tail (`cascade_and_reopen`), the +fail-loud conversion of the storage decode layer, and the +**do-not-simplify list**, which now lives beside the invariants it protects +([`docs/invariants.md`](../invariants.md), "Do-not-simplify"). Open +remnants and deliberate declines-with-reasons are in +[`register.md`](register.md). diff --git a/docs/review/2026-06-10-test-coverage.md b/docs/review/2026-06-10-test-coverage.md new file mode 100644 index 00000000..9ceb77d6 --- /dev/null +++ b/docs/review/2026-06-10-test-coverage.md @@ -0,0 +1,15 @@ +# Test-coverage review (2026-06-10) + +Third companion to the correctness review: what the suite pins, what it +fails to pin, and which harness levers exist or are missing. **Verdict:** +recovery is the best-tested subsystem (the full dispatch matrix at unit and +e2e level, libfaketime mid-run clock jumps, respawn loops, TCP-proxy outage +injection, Anvil mempool control); the one structural hole was the duality +having no direct mechanism — closed since by the watchdog non-genesis +byte-compare e2e (sequencer-produced batches through the real guest +machine) and the I1 predicate-vs-fold agreement table. + +The owed-test list, remaining harness levers, and accepted-untested items +live in [`register.md`](register.md) ("Owed tests"), with statuses verified +2026-08-22. Decision recorded there: do not resurrect a TEST_PLAN scenario +matrix — the dated, finite owed list is the artifact. diff --git a/docs/review/2026-06-25-cockroach-recovery-rooting.md b/docs/review/2026-06-25-cockroach-recovery-rooting.md new file mode 100644 index 00000000..324529a3 --- /dev/null +++ b/docs/review/2026-06-25-cockroach-recovery-rooting.md @@ -0,0 +1,18 @@ +# Settled design — cockroach-recovery batch-tree rooting (2026-06-25) + +Design session (with an adversarial panel) for how `setup --recovery` roots +the rebuilt batch tree at the resume nonce `N'`. + +**Decision: the `batch_tree_anchor` singleton** — the parentless root carries +the anchor nonce (0 for genesis, `N'` for recovery), validated exactly by the +contiguity trigger and frozen once setup completes. No sentinel batch row. +The design now lives in [I16](../invariants.md) and +[`docs/recovery/cockroach.md`](../recovery/cockroach.md); the rejected +sealed-sentinel alternative and the "`N` is trusted — no recovery-time +verifier" resolution (the proposed cross-check was circular) are recorded in +[`register.md`](register.md). + +The follow-up e2e round trip caught a real bug — the content-identity check +self-diverged during recovery against the empty rebuilt tree — fixed by the +anchor-aware frontier (recorded on I15) with frontier population deferred to +`run`'s first sync. diff --git a/docs/review/2026-06-26-branch-deep-review.md b/docs/review/2026-06-26-branch-deep-review.md new file mode 100644 index 00000000..45ca7742 --- /dev/null +++ b/docs/review/2026-06-26-branch-deep-review.md @@ -0,0 +1,20 @@ +# Deep branch review — setup/run split → cockroach recovery (2026-06-26) + +Whole-branch review of the setup/run split, scheduler-library extraction, +fold engine, and `setup --recovery`, plus a follow-on multi-agent adversarial +sweep. **Eight findings confirmed, all fixed; three refuted.** + +The lasting outcomes live elsewhere: the recovery spec is +[`docs/recovery/cockroach.md`](../recovery/cockroach.md); the authoritative +protocol contracts written during this review are +[`docs/protocol/scheduler-semantics.md`](../protocol/scheduler-semantics.md) +and +[`docs/protocol/application-contract.md`](../protocol/application-contract.md); +per-finding dispositions and the deferred-with-reason items (e.g. the +`FoldInputSource` abstraction, declined as over-abstraction over a single +call site) are in [`register.md`](register.md). + +Notable fixed findings, for the record: partial-recovery re-anchoring and +genesis-over-residue guards; the recovery drain capped at `C` so `(C, H1]` +deposits are led exactly once by `run`; wrong-chain RPC during recovery sync +classified terminal instead of restart-looping. diff --git a/docs/review/2026-08-01-containment-adr-review.md b/docs/review/2026-08-01-containment-adr-review.md new file mode 100644 index 00000000..5167d51a --- /dev/null +++ b/docs/review/2026-08-01-containment-adr-review.md @@ -0,0 +1,16 @@ +# Containment ADR review (2026-08-01) + +Two review rounds on the terminal-containment cutover branch. The +architectural turn was accepted, then re-evaluated with the maintainer +(2026-08-02): the unimplemented `LiveKernel`/reader-mailbox design was +rejected on the actual completeness/cost boundary — the content-identity +check is a narrow backstop, not a divergence oracle, and SQLite remains the +durable coordination plane. + +Every mechanism this review shaped now lives in the +[authority-boundary ADR](../plans/2026-08-authority-boundary-adr.md) +(including the rejected `RunEpoch`/`EffectGate`/`LiveKernel` alternatives and +the accepted divergence-window bound recorded on +[I15](../invariants.md)); the intermediate marker-file protocol it hardened +was later deleted wholesale. Per-finding dispositions are in +[`register.md`](register.md). diff --git a/docs/review/2026-08-18-over-engineering-review.md b/docs/review/2026-08-18-over-engineering-review.md new file mode 100644 index 00000000..f5c472de --- /dev/null +++ b/docs/review/2026-08-18-over-engineering-review.md @@ -0,0 +1,45 @@ +# Over-engineering review (2026-08-18) + +Full-branch review of the authority-boundary + durable-history-foundation +branch against the project's design goals (readable, auditable, every +mechanism judged against its weight): seven parallel subsystem reviews, each +proposal adversarially cross-examined against the invariants register, the +TLA+ models, and git history, plus an independent premise challenge of the +ADR. 141 mechanisms inventoried: 98 keep, 25 simplify, 6 cut, 12 question. +This review adopted the calibration rule now in AGENTS.md (the complexity +budget belongs to concurrency, durability, and hostile-L1 robustness). + +**Verdict: not over-engineered — unevenly engineered.** The ADR's premise +survived attack; the rejected alternatives left no residue in code. Three +findings cut against "smallest sufficient design": the in-process containment +predicate was still convention at ~11 hand-placed sites (fixed by the +`Authorized` token), ~700 lines of mechanical repetition had accumulated +(harvested), and the lifecycle module implemented the right guarantee with a +heavier representation than needed (re-platformed, then removed — see below). + +**Outcomes** (all landed 2026-08-18/19, each wave adversarially re-reviewed +post-commit; per-item dispositions in [`register.md`](register.md)): + +- Eleven defects fixed (fail-open classification arms, admission bypasses, + containment publication window, snapshot-route gating, typed app-boundary + refusals). +- ~700-line harvest: worker-exit plumbing collapsed with terminality beside + each error type; a real `RuntimeScope` with `ShutdownSignal` reduced to its + name; fee-oracle bootstrap behind its module; recovery type-stack trimmed + to the one `RecoveryProgress` enum; dead surface deleted. +- The `Authorized` externalization token: the containment consult became a + compile-time obligation of the effect functions. +- Module homing: command *brackets* to `commands/` (with their config/error + taxonomy, `RunError` → `CommandError`), the capability substrate alone in + `runtime/`, `L1Config` to `l1/`, the clock to the crate root. +- **Lifecycle arc:** re-platform to singleton + audit trail (decision L1) → + admission gating removed entirely, facts govern (decision L2, 2026-08-19) + → journal narrowed to the terminal-fault black box (decision L3, see + [`2026-08-22-lifecycle-simplification.md`](2026-08-22-lifecycle-simplification.md)). + The surviving design rationale lives in the ADR and the invariants check + policy. + +Still open from this review: the 500 ms latency contract does no design work +(nothing is shaped by it — decide what it is *for*), and the +catch-up ACK-latency measurement owed to the benchmark harness. Tracked in +the register. diff --git a/docs/review/2026-08-22-lifecycle-simplification.md b/docs/review/2026-08-22-lifecycle-simplification.md new file mode 100644 index 00000000..a4f363fc --- /dev/null +++ b/docs/review/2026-08-22-lifecycle-simplification.md @@ -0,0 +1,60 @@ +# Lifecycle simplification review — L3 (2026-08-22) + +Fresh-eyes review of what remained of the lifecycle machinery after decision +L2 (admission gating removed, 2026-08-19). Method: two exhaustive read-only +sweeps (a 28-class terminal-fault re-detection map; a journal +weight-and-consumers audit), a six-refuter adversarial verification of every +load-bearing claim before acting, and a fifteen-agent post-landing review of +the diff. + +**Findings that grounded the decision:** + +- The L2 characterization was true: admission was exactly three facts, and + the attempt journal had zero production reads for decisions — its ~490 + production lines across ten files bought only what tracing already + provided, plus the terminal-cause row. In `admission.tla` the journal + variable was a bijective ghost of the controller (removing it left TLC + state counts byte-identical). +- The "terminal faults refuse at re-detection, not at boot" trade is far + narrower than it sounds: everything with durable or deterministically + re-derivable evidence re-refuses before the first soft confirmation, and + the batch/frame spine is re-read within seconds of launch. The verified + residual (cold payload bytes below the lane checkpoint, reachable only via + the WS catch-up window or a pending batch's re-encode; faults with no + durable evidence at all) is recorded as an accepted boundary in the threat + model. + +**Decision L3 (landed):** the journal narrowed to the `terminal_faults` +black box — append-only command+cause rows, best-effort. Principle adopted, +now in the invariants check policy: **telemetry writes are verdict-neutral** +— they sat on the brackets' `?` paths and could change exit codes. +`admit_runtime` collapsed to one consistent inspect + reduce; `RunId` and +the event vocabulary went with the settle plumbing. No boot machinery was +added in the journal's place: neither a durable verdict gate (it needs an +acknowledgement to exit, which carries no information the reducer doesn't +re-derive) nor a boot-time full-integrity sweep (expensive, and blind to +semantic violations outside its read set). + +**Verdict-integrity defects fixed with it** (each adversarially confirmed +first): settle-masking (a settle-step failure could replace a terminal +verdict with exit 1 — and settle was the *sole* exit-code determinant for +most terminal paths); `CommandError::Lifecycle` classified wholesale +terminal (a transient `SQLITE_BUSY` on a lifecycle write paged); signer +misconfiguration classified as unclassified I/O in all three keyed commands. +The misconfig-poison taxonomy question from the 2026-08-18 review closed +with L3: there is no poison to apply; what remained was exit-code accuracy. + +**Post-landing review (8 confirmed / 4 refuted):** one real regression — the +Ok-path divergence refusal had been dropped with `settle_clean`, letting a +clean drain over freshly persisted divergence exit 0 (the one code that +breaks the supervisor's restart-then-refuse rediscovery chain); restored as +an explicit fact check and test-pinned. One missing test pin added; six +doc-staleness items fixed. A claimed re-detection gap +(`finalized_snapshot.inclusion_block` NULL at boot) was refuted — the column +is `NOT NULL` at the engine; no boot assert was added. Lesson recorded: when +deleting a mechanism, sweep for its *vocabulary* with review agents, not a +bare grep — one grep pipeline silently returned empty on files that +contained the pattern. + +All landed in the squashed branch commit; per-item dispositions in +[`register.md`](register.md). diff --git a/docs/review/2026-09-03-branch-stocktake.md b/docs/review/2026-09-03-branch-stocktake.md new file mode 100644 index 00000000..9f3afb22 --- /dev/null +++ b/docs/review/2026-09-03-branch-stocktake.md @@ -0,0 +1,688 @@ +# Branch stock-take (2026-09-03) + +Stock-take of the authority-boundary branch (PR #28, seven commits on +`f59ec25`) before it leaves draft, asked as: what is this branch for, is it +over-engineered, what next. Method: first-hand reads of the runtime, command, +recovery, lifecycle, and history code; then a read-only fleet of seven +subsystem lenses and five premise challengers (threat model, minimal design, +refuted-list audit, CI root cause, roadmap); then three adversarial refuters +per proposal for the eighteen highest-ranked proposals. Every proposal the +fleet raised is recorded here, including the ones that were not put to a +jury. Per-item dispositions that are actionable now live in +[`register.md`](register.md) (open findings 19–30 and the 2026-09-03 refuted +block); this ledger is the full record and will be distilled when it closes. + +**How to read the status tags.** + +- **confirmed** — put to three refuters; at most one refuted it. The recorded + text includes the jury's amendments, which are load-bearing. +- **refuted** — put to three refuters; at least two refuted it on code truth + or on a registered invariant. The reason is recorded so it is not + re-proposed without new evidence. +- **unverified** — raised by one reviewer and not adversarially checked. + Treat as a reviewer's claim: re-verify the cited lines before acting. +- **verified first-hand** — checked directly against the tree during the + stock-take, independent of the fleet. + +## Verdict + +Proportionate overall, with three named pockets of residue. The maintainer's +fear was quantitatively wrong about scale and right about residue. + +| What | Lines | +|---|---| +| Branch diff | +20,459 / −5,252 across 127 files | +| Authority machinery changed (process lock, scope, lifecycle facts, supervisor) | ~1,700, under 900 production | +| Share of the branch | ~8% | +| Test functions | 418 → 588 | + +The lifecycle machinery that felt over-built was built and removed inside the +branch (decisions L1 → L2 → L3, ~490 production lines); what survived is three +admission facts and one append-only table. The heavy parts of the sequencer +are recovery and storage, which predate the branch and defend in-scope L1 +outage and zombie-transaction threats. + +The three pockets, in descending confidence: + +1. **The black box's write path.** `terminal_faults` has zero production + readers; the in-scope recorder opens a second SQLite writer from inside + containment, is the reason the "arm the watchdog before recording" ordering + hazard exists, and a contained run that drains normally writes two rows. +2. **The exit-code test encoding.** Sixty-seven hand-built projection asserts + pin a pure function four times over, while no test asserts that a real + failing process exits with the promised code; renumbering the terminal code + passes the whole suite. +3. **Documentation fan-out.** Fact-derived admission is described in fourteen + places and the divergence freeze in eleven files; the branch's own L3 + rename missed three "journal" sites. + +Beyond mechanisms, four lenses independently found one defect class: prose +that claims more than the types enforce (see finding 25 in the register). + +**What passed the weight test and should not be cut:** the process lock; the +containment bit and the `Authorized` token at the ack, L1 send, and WS emit; +the pure reducer over one consistent inspection; the `RuntimeAdmission` +witness; the two-second abort watchdog (`/livez` returns 200 unconditionally, +so on a wedged post-containment drain nothing else pages); the clean-exit +divergence re-check. + +## Verified first-hand + +- CI red cause: `tracing-subscriber` 0.3.23 enables ANSI whenever `NO_COLOR` + is unset, with no TTY check (`fmt_layer.rs:743`); the harness inherits the + parent environment and pins only `RUST_LOG`; the e2e assertion at + `tests/e2e/src/test_cases.rs:3151-3157` greps for `status=TipInDanger(` and + the log carries `ESC[3mstatus ESC[0m ESC[2m= ESC[0m TipInDanger(0)`. It is + the only log-grep assertion in the suite. Both wallet-sequencer mains lack a + `with_ansi` call. +- PR metadata is stale: title "Fix storage decode policy", head branch + `feature/review-ledger-and-tracks`, body describing the decode-policy scope + and citing a retired codename; no risk/compatibility paragraph although the + baseline migration is rewritten in place and the `Application` hooks are + renamed. No reviewer comments. No `TODO`/`FIXME`/`unimplemented!` in the diff. +- `/livez` returns 200 unconditionally (`egress/api/health.rs:41-43`). +- `finalized_state` handles an impossible `None` on the `NOT NULL` + `inclusion_block` by escalating to containment (`egress/api/snapshot.rs:139-146`). +- Stale vocabulary: "journal" at `storage/history.rs:201`, + `commands/run/workers.rs:400` and `:680`; an "acknowledge" command at + `commands/error.rs:11`; `docs/recovery/README.md` names + `DangerDetectorExit::DangerDetected` (the type is `WorkerExit::DangerDetected`). +- `RecoveryProgress` derives `Copy`; `docs/recovery/README.md:315`, + `admission.tla:23`, and `recovery/mod.rs:893` call the witness "non-clone". +- `Authorized` is a real signature obligation at three functions + (`submit_batches`, `acknowledge_included`, `send_authorized`); it is minted + and discarded at `snapshot.rs:104/129/182`, `ingress/api.rs:87`, and + `inclusion_lane/mod.rs:211`; the batch-close and reconciliation commits at + `inclusion_lane/mod.rs:165/309` use the raw predicate. +- `terminal_faults` has zero production readers; a contained run appends two + rows (recorder raw cause, then the bracket's prefixed cause); + `latest_terminal_fault` returns the second. +- `ShutdownSignal` on main was 43 lines; `runtime/shutdown.rs` is now 450. + `http.rs` grew a lease-release supervisor with two containment call sites. + +## Confirmed (jury) + +- **cfg-test-gate-ensure-open-tip** (3–0). `Storage::ensure_open_tip` + (`storage/ingress.rs:104`) is `pub` with zero production callers — a new + instance of open finding 17 created by this branch, sitting beside its + guarded replacement. Gate it `#[cfg(test)] pub(crate)` (not private: eight + of nine callers live outside `storage::ingress`), de-link the two intra-doc + references at `ingress.rs:118` and `:486`, correct `:486`'s claim that the + runtime calls this form, and fix `docs/snapshots/lifecycle.md:26-31`, which + still credits it with the production genesis Tip. +- **stale-decision-carries-the-failed-condition** (3–0). + `ensure_open_tip_for_recovery` (`storage/recovery.rs:254-259`) raises + `StaleDecision { expected: Safe, actual: facts.danger }` for a disjunction, + so the `has_open_tip` case renders "expected Safe, found Safe", and + `recovery_tests.rs:114-132` pins that as intended. Add a payload-free + `RecoveryMutationError::TipAlreadyOpen`, split the two checks, add a paired + `RecoveryRetryReason::TipAlreadyOpen` so `classify_mutation` (which today + discards `expected`) carries it to the operator, classify Retry, update the + test and the polarity pin. Roughly +14/−6 across three files. The arm is + production-unreachable under the process lock; this is diagnostics. +- **detector-takes-shutdown-signal-not-runtime-scope** (2–1). + `DangerDetector` and `InputReader` use the scope for exactly one thing, + `wait_for_shutdown`, and each already carries a construction-required + `ProcessLock`. Narrow `start`/`start_preflighted`/`run_forever` to + `ShutdownSignal` and pass `scope.signal()` at the two launch sites. The + change is incomplete without restating three doc comments that assert the + property it relocates: `workers.rs:100-105` and `:1120-1124` ("every spawned + worker retains a RuntimeScope clone, which also retains the process lock") + and `shutdown.rs:96-99` ("workers that touch the data directory take a + scope", already false for the submitter). Restate as: workers that + externalize or contain take a scope; data-directory ownership is a separate + construction-required `ProcessLock`. The dissent would narrow only the + detector. +- **app-with-progress-wrapper** (2–1, a do-not-adopt). Moving + `ApplicationProgress` into a sequencer-owned wrapper (deleting both + capabilities, the seal, three trait methods, all three asserts, ~130 lines) + is not viable: the pair is inside the canonical SSZ bytes + (`examples/app-core/src/wallet_snapshot.rs:41-42`) that `create_dump` writes, + `/finalized_state` streams, and the watchdog byte-compares, and the canonical + machine advances it inside its own state transition; cockroach recovery reads + the clock out of a dump into a wiped database (`commands/setup/mod.rs:433-451`, + `Checkpoint::load`). The rationale for the clock exists at + `docs/snapshots/format.md:113-118`; the missing piece is the composition. + Add one sentence to `docs/protocol/application-contract.md` §4 and to + `ApplicationProgress`'s doc comment, cross-referencing `format.md`, and + extend `format.md`'s "must live in the canonical state bytes" sentence to + cover `executed_input_count`. +- **bound-or-prove-drive-recovery-termination** (2–1). `drive_recovery` + (`recovery/mod.rs:288-313`) is an unbounded loop with one cycle: + `Repaired` + `Safe` + `!has_open_tip` → `EnsureOpenTip` → `Repaired`. No + watchdog exists on the boot path (the scope is constructed in `prepare`, + after recovery). Main could not spin. Take the postcondition, not the loop + bound: after `open_fresh_tip_in_tx` in `ensure_open_tip_for_recovery`, + re-read `has_valid_open_batch` and return a new typed variant classified + `RecoveryError::refuse` (exit 30) — not a `debug_assert` (compiles out in + release; the file's existing postcondition at `ingress.rs:538-541` is one), + and not `StaleDecision` (maps to retry and relocates the non-termination into + the supervisor). Record the ≤5-phase bound in `drive_recovery`'s doc. + Alternative accepted by two jurors: make the reducer's `Repaired`+`Safe`+no-tip + arm a terminal `Refuse`, removing the cycle from the pure function. The + dissent notes the antecedent is unreachable by SQLite semantics; the + counter-argument that carries is fidelity — `admission.tla:271-286` + hardcodes `hasOpenTip' = TRUE`, so TLC proves termination of a model whose + postcondition the code does not enforce. +- **typed-key-source-io-error** (2–1). `resolve_key_source` + (`commands/config.rs:243-254`) returns a bare `std::io::Error` that lands in + `CommandError::Io` → exit 1 ("restart with backoff") for a missing or + unreadable key file, while bad key content in the same file exits 30 via + `SignerMisconfig`. Kind-filter rather than blanket-map, mirroring + `referenced_artifact_io_is_terminal` (`dump_info.rs:49-58`): NotFound, + PermissionDenied, InvalidData, IsADirectory, NotADirectory terminal; + everything else operational (a not-yet-mounted secret must not consume the + do-not-restart code). Prefer a distinct `BootstrapError::KeySourceUnreadable + { path, kind }` over reusing `SignerMisconfig`; never echo file contents. + Roughly +30–40 lines with the predicate and tests. Consider the same + treatment for `create_dir_all` or record why not. +- **table-drive-exit-code-tests** (2–1). Fold the five per-class tests and + the duplicating verdict test (`error.rs:782-828`) into one `const CASES` + table keeping every distinct error shape and every reason string as the + assert message; drop the verdict column (derivable from the bijection); + assert `is_terminal()` per row, which extends a five-shape pin to ~54. Do not + invent rationales for rows that carry none. Realistic saving is 110–150 + lines, not 250. The point is the spend: no test asserts a real failing + process's exit code (the four failure-path e2es assert only `!success()`), + and the `EXIT_*` values appear as literals only at their declarations, so + renumbering `EXIT_TERMINAL` to 31 passes the suite. Add SIGTERM → 0 (the + harness already waits and discards the status) and one 30-class failure → 30 + (`run` on a never-set-up data directory needs no new lever), asserting integer + literals. Correction from the jury: composition is pinned in-crate at + `workers.rs:1209/1228`, `run/mod.rs:237`, `startup_hygiene.rs:168/191`, + `commands/mod.rs:330`, `process_lock.rs:159`; the gap is the process-level + projection at `harness.rs:117-119`. +- **dedupe-terminal-fault-rows** (2–1 for documenting; 0–3 against skipping + by variant). Document the two-row shape at `record_terminal_fault`, in the + ADR's black-box paragraph, and in the runbook's postmortem line, including + the asymmetry: a clean contained drain yields two rows; a controller panic or + watchdog abort yields one. Do not skip the bracket write when the error is + `StorageInvariantViolation`: the recorder swallows both `open_writer` and + `record_terminal_fault` failures into a `warn!`, so the variant does not + prove a row landed, and the post-drain bracket write is the attempt more + likely to succeed after `SQLITE_FULL` or contention (5 s `busy_timeout` + against a 2 s abort deadline). If one row per fault is wanted later, + condition the skip on evidence (an `AtomicBool` the recorder sets on + success), not on the variant. + +## Refuted (jury) — do not re-propose without new evidence + +- **supervise-workers-with-joinset** (3–0). `select_first_exit` and `finish` + deliberately read the same worker return two ways: `WorkerStop::from_select` + maps `Ok(Ok(()))` to `StoppedUnexpectedly` (the runtime is live), while + `from_shutdown` maps it to `Ok(())`. Feeding both phases from one `JoinSet` + of `wait_for_*_shutdown` futures collapses them into the shutdown reading, so + a worker that dies silently while live yields a value `FirstExit` cannot + represent; the available completions are "run with a dead lane" or "drain + and exit 0", the one code `run/mod.rs:82-87` names as breaking the + supervisor's rediscovery chain. `into_supervision(self)` would also drop + `ShutdownOnDrop`, requesting shutdown milliseconds after launch; the + conditional fee-oracle push relocates rather than deletes; and + `FirstExit::detector` plus its mapping tests disappear (one of the four + reasons the register already refuted this shape on 2026-08-23). **Survives:** + the `swap_remove` hazard at `workers.rs:653-655` is real and unwritten in + types; a cleanup-only `JoinSet` built inside `finish`, with the live race + untouched, was not what the jury examined. +- **collapse-preparedruntime-into-boot** (3–0). The load-bearing claim ("zero + tasks during fallible work is reviewer-visible, not type-enforced") is false: + `fn launch(self, _admission: RuntimeAdmission) -> Workers` (`workers.rs:288`) + is non-async and non-`Result`, so a `?` or `.await` between admission and the + six spawns is a compile error today. `async fn boot(..) -> Result<..>` makes + both silently legal, reopening a guarantee registered in the check policy, + ADR mechanism 1, and AGENTS.md, and the linearization argument at + `recovery/mod.rs:477-483`. Under `boot` nothing consumes `RuntimeAdmission` + (`let _a = admit_runtime()?` satisfies `#[must_use]`). Commit 02a2b34 ran + this pass and deliberately stopped here. **Survives:** the test + `preparation_outliving_clean_facts_cannot_launch` (`workers.rs:1161`) never + calls `launch`; rename it to what it asserts. +- **make-authorized-token-uniform** (3–0). `LeasedDumpBody` does not exist + (the primitive is `stream_body(file, guard)` at `snapshot.rs:214`); + `finalized_inclusion_block` (`snapshot.rs:101-120`) has no streaming + primitive to receive a token; and `ingress/api.rs:87` is not a pre-check — + its comment calls it the publication gate, it runs after the lane's ack + resolves, and it immediately precedes the success body that is the soft + confirmation leaving the process, the ack family the ADR names as a token + site. Only the `inclusion_lane/mod.rs:211` half survives (a fast-turn entry + gate; the real ack boundary re-consults at `:248`). **Survives:** the doc + tightening — the token proves "consulted at some point in this borrow", not + "at this effect boundary" (the poster mints at `worker.rs:243` and then does + a chain-id RPC, fee estimation, and a nonce fetch before its own re-checks); + and the coverage claim in the ADR/register should say three compile-forced + primitives plus hand-placed consults at the HTTP 200 gate and the two lane + mutation commits, or the 200 body should take the token (~8 lines). +- **recovery-polarity-unconstructible** (3–0). Diagnosis exact: + `RecoveryError::retry(RecoveryRefusalReason::CanonicalDivergence{..})` + compiles today and would project the absorbing refusal to exit 20 (bounded to + one restart by the next boot's preflight). But `recovery` is `pub mod` and + both enums have public variants, so deleting the `#[from]` impls removes the + shortest spelling, not the route: `RecoveryError::retry(RecoveryFailure::PolicyRefusal(r))` + still compiles, and that longer spelling is the dominant idiom at all 15 call + sites. Cost ~16 renames for a property not achieved. The invoked precedent + (1fcb9aa) deleted the violating value from the type; this does not. +- **single-table-per-error-type** (3–0). `From + for BootstrapError` (`error.rs:671-683`) performs no terminal/transient + decision — it selects among three variants with distinct fields, and the + verdict is taken later over the `BootstrapError` taxonomy, whose variants + have four other producers. "Have both sites read one `is_terminal`" is not + implementable; the result is a third table. Part (b)'s premise is false: + `reader.rs:96-104` states the phase-dependence for `Bootstrap` and `Join`. + Renaming to `is_terminal_in_worker` is wrong for `FlushError` (no + `WorkerExit` arm), and "phase in the type" is blocked because the phase + belongs to the caller (`create_provider(..).map_err(InputReaderError::Bootstrap)` + appears identically in `sync_to_current_safe_head` and `run_loop`). + **Survives:** `classify_input_reader` (`recovery/mod.rs:526`) carries no doc + comment and its `Bootstrap`/`Join` refusals are pinned by no test; and a + pre-v3 InputBox exits 1 under `setup` but 30 under `run` — a separate + finding. +- **flatten-recovery-error-to-one-enum-with-is-retryable** (3–0). A flat + `is_retryable(&self)` must be total over the value, and one variant carries + two verdicts: `ProductionRecoveryDriver::flush` (`recovery/mod.rs:429-438`) + maps `VerifiedSignerProviderError::ChainIdRpc` → retry and `::Create` → + refuse into the same `RecoveryFailure::Provider(String)`. Either resolution + regresses (a bad RPC URL restart-loops forever, or a transient chain-id + timeout pages), nothing pins either arm, and dropping the `Box` risks the + deliberately managed `CommandError` footprint under `result_large_err`. + **Survives:** split `Provider(String)` into two verdict-determined variants + as an independent fix; the wrapper's doc claims a context-sensitivity the + `classify_*` functions do not use. +- **drive-recovery-owns-phase-to-progress-mapping** (2–1). The replacement + is not total: `(RecoveryPhase::Flush, PhaseOutcome::Done)` has no target + because `Flushed { observed_safe_block }` needs a block number the loop does + not hold; the pre-existing implementation of exactly this mapping + (`PhaseCompletion` + `transition_after_phase`) was deleted by `ed41f9b`, whose + message pre-answers the argument. **Survives:** delete + `RecoveryDriver::admitted` (`recovery/mod.rs:283`, a production trait method + whose only implementor pushes a string into a test trace; `drive_recovery` + returns `Ok(())` only from the Admit arm); and the five trace tests exercise + the double's copy of the mapping while production's copy is pinned by no unit + test. +- **merge-stringly-bootstrap-variants** (2–1). `FeeOracleMisconfig` has two + further producers (`setup/mod.rs:144` and `:171`) passing bare strings whose + "fee oracle misconfiguration" words exist only in the variant's Display, and + the black box stores `error.to_string()`, so merging attributes an operator's + Uniswap mistake to a trusted-code fault in the one postmortem artifact. Part + (b) demotes a compile-forced classification to an unchecked `&'static str` + discriminant, the shape the register refuted twice on 2026-08-23. Real delta + ~−15, not −25. +- **terminality-trait-for-workerstop** (3–0). Inverts its goal: + `WorkerExit::is_terminal` already matches all seven variants by name; the + trait admits `fn is_terminal_invariant(&self) -> bool { false }` exactly as + plausibly as `|_| false`. `impl TerminalityOf for std::io::Error { false }` + installs a crate-wide answer for a type the codebase has decided has no + context-free answer (`dump_info.rs:49-58` classifies several kinds terminal); + a `pub(crate)` trait in a public type's bound trips `private_bounds` under + `-D warnings`; and `is_terminal_invariant` is inherent on eight types, only + five of them `WorkerExit` payloads. Delta inverts to +4..+15. +- **prune-duplicate-tla-actions-and-run-check-admission-in-ci** (3–0). + "Nix already provides tlc" is false for CI: no Nix expression or `.envrc` is + tracked, `ci.yml` provisions tools by hand, and `just` is absent from the + `rust` job; TLC also checks the spec against itself and says nothing about + spec-vs-Rust drift. The three deletions are state-space-neutral (`Crash` + subsumes every settle action), but the rationale is false: `decision` + records Retry/Refuse, `DecideRetry`/`DecideRefuse` have distinct guards, and + `InspectRetry` encodes its own comment ("a known local divergence cannot be + masked by the retry edge"). **Survives:** put the 860-state model under CI as + a properly pinned standalone `formal` job (JDK + `tla2tools.jar` pinned by + version and sha256 in `toolchain-pins.env`). + +## Unverified (raised by one reviewer, not put to a jury) + +Re-verify the cited lines before acting on any item below. + +### Runtime authority and containment + +- **drop-in-containment-fault-recorder** (threat challenge, Δ−110). Delete + the `FaultRecorder` alias, field, `set_fault_recorder`, and the recorder + invocation from `runtime/shutdown.rs`; delete `install_terminal_fault_recorder` + from `workers.rs`. Containment becomes three non-blocking steps (set the + cause, arm the watchdog, request shutdown), removing the "either may block" + ordering hazard and the second SQLite writer inside containment. The bracket + write at `run/mod.rs:90` keeps recording every contained fault that settles. + Loss: a fault whose drain hangs past 2 s and exits via SIGABRT leaves no row + — already the documented status for unclean deaths + (`docs/watchdog/operator-deployment.md:392-395`). Note the confirmed + dedupe verdict above: the bracket write is a genuine retry, which argues for + keeping one writer rather than two, and for the bracket one. +- **delete-terminal-faults-black-box** (threat challenge, Δ−330) and + **cut_black_box** (minimal design, Δ−185). Delete the table, its two + triggers, `TerminalFault`, `record_terminal_fault`, `latest_terminal_fault`, + `LifecycleCommand::parse`, `record_terminal_fault_best_effort` and its four + call sites; replace the runbook's `SELECT * FROM terminal_faults` paragraph + with the log-and-exit-code instruction. New evidence against decision L3: + zero production readers; no CLI or API surface; the write path spans four + files; cockroach recovery wipes the database in exactly the incident class + where the postmortem matters. Counter-argument neither author could dismiss: + a Kubernetes Deployment restarts regardless of exit code, so a terminal + fault restart-loops and the first cause could rotate out of the logs while + the black box retains it. Judgment call, not a defect. +- **close-or-correct-the-token-coverage-claim** (threat challenge, Δ+8). + Either make `ingress/api.rs:85-92`'s success response take `Authorized` + (mirroring `acknowledge_included`), or amend ADR mechanism 1 and the + register's settled entry to the true scope. Do not extend the token to the + lane's mutation commits (they sit inside `&mut self.storage` borrows). The + jury's refutation of the "uniform" proposal above endorses this framing. +- **sketch_boot_shutdown** (minimal design, Δ−260) — a reference sketch that + keeps the lock, scope, token, `ShutdownOnDrop`, reducer, hygiene, lifecycle + facts, and the typed `Workers`/`FirstExit`, and deletes `ShutdownSignal` + (fold into the scope), `RuntimeAdmission`, `PreparedRuntime`/`WorkersConfig`, + and the `WorkerId`/`ComponentShutdown`/`next_component_shutdown`/six-waiter + drain in favour of `Option` fields taken at the winning select + arm and one `tokio::join!` over a generic `drain`. Partly overtaken: the + jury refuted the `PreparedRuntime` collapse and the `ShutdownSignal` half is + contradicted by the confirmed narrowing (the slim half gains two consumers). + The `Option`-take drain is the one part not examined by a jury; it avoids + all four grounds of the 2026-08-23 refutation (named fields, named arms, no + `swap_remove`, no empty-list race). +- **cut_admit_runtime** (minimal design, Δ−60). Delete `RuntimeAdmission`, + `admit_runtime`, `AdmissionChanged`, and the `launch(_admission)` parameter; + keep the fallible-then-infallible ordering. Argument: between the reducer's + `Admit` and launch, the lock excludes every other process and zero tasks + exist, so only wall-clock drift can change the facts, and those arms are the + Retry class re-derived by the detector within one 2 s poll. Adjacent + refutation applies in part: the jury defended the witness as the thing that + makes "launch only from a fresh admit" a compile fact and the basis of the + linearization at `recovery/mod.rs:477-483`. Low priority. +- **taxonomy_min** (minimal design, Δ−145). Regroup `BootstrapError`'s + seventeen variants into verdict-uniform groups (`Misconfig(..)` uniformly 30, + `Transient(..)` uniformly 20, `Recovery`, `SetupNotComplete`, `SetupRefuse`, + `OpenStorage`), move `IdentityError::FirstBootRequiresL1` into the transient + group so `IdentityError` becomes uniformly terminal, delete + `CommandFailureVerdict` (five variants in bijection with five constants, two + consumers), delete `WorkerId` if the drain no longer needs identity. The + taxonomy has exactly one consumer outside the crate (`harness.rs:117`). + Partly overtaken: the jury upheld keeping `is_terminal()` as the thing the + black-box write gates on. +- **leased_dump_inclusion_block_non_optional** (refuted-list audit, Δ−4; + premise verified first-hand). Stop sharing `LeasedDump` between the + finalized and latest lease queries (own return type or `LeasedDump`), and + delete the `let Some(inclusion_block) = leased.inclusion_block else { + contain(..) }` branch at `snapshot.rs:139-146`. The column is `NOT NULL` + (`0001_schema.sql:855-856`); the L3 review refuted a boot assert on it and + left a heavier runtime branch that maps a type artifact to exit 30, contrary + to the check policy's "no `Option`-handling for can't-be-`None`". +- **startup_log_last_terminal_fault** (refuted-list audit, Δ+8). After the + preflight in `run`, read `latest_terminal_fault` and emit one `warn!` when + present. Explicitly not a gate: no acknowledgement, no branch on the value. + The recorded refutation argues against a gate on a verdict; a read that + changes no decision is untouched by it. This would give the black box its + first in-product reader; if declined, say in the register that the black + box is an out-of-process artifact by design. +- **startup_hygiene_single_finalized_read** (refuted-list audit, Δ−6). + `require_finalized_snapshot` and `restamp_finalized_promotion` each query + `finalized_dump()`; fetch once in `run_snapshot_hygiene` and pass the row. + Ordering of the five steps is unchanged. +- **reconsider-release-supervisor-weight** (lane lens, ~175 lines). The + supervised lease-release queue in `http.rs:157-235,325` (unbounded MPSC of + boxed closures, a `JoinSet` supervisor with a two-armed `select!`, a drain + awaited inside `axum::serve`, containment on both joins, `ReleaseScheduler` + changed to `Arc` plus a second reporter) defends a real but narrow + hole: a `StatementChangedRows` on release means the leased row vanished, + which nothing else re-detects. Two lighter shapes: (a) keep the + classification, drop the supervisor, accept that a release racing the very + end of shutdown may miss classification (~−120 lines); (b) keep the drain + but move it to the egress snapshot module that owns leases, so `http.rs` + stops hosting a runtime component. The queue is unbounded and bounded only + by concurrent snapshot requests, which have no cap. +- **drop-token-ceremony-at-bool-sites** (lane lens, Δ−10). Overlaps the + refuted uniform-token proposal: the `/tx` publication-gate half is refuted + (it is the ack); the snapshot half survives only as "push the token into + `stream_body`" for the two streaming routes. Separately worth a maintainer + decision: the `/tx` gate returns 503 for an operation that is durably + committed and may still reach L1; the API contract should state the + client-visible semantics of "503 after commit". + +### Error taxonomy + +- **fee_price_stamp_surface_or_drop** (refuted-list audit, Δ+10). + `log_gas_price_updated_at_ms` is written on every refresh + (`storage/fee_oracle.rs:29-40`) and read only by tests, yet the threat model + cites it as the honest telemetry that justifies having no expiry gate. + Either surface the age in `GET /healthz` as an informational field, or + include `retained_price_age_ms` in the existing transient-refresh warn, or + drop the column and fix the threat-model sentence. No threshold, refusal, or + lifecycle effect is proposed. +- **register_provenance_and_wording** (refuted-list audit, Δ+6). Applied in + the register on 2026-09-03: the fee-price-age refutation was added by + 143a290 (2026-08-25) but filed under "From the ADR re-evaluation + (2026-08-01/02)"; the boot-gate refutation's "(in any form)" is broader than + the argument it rests on; the drain-merge entry says "Scope-narrowed + 2026-08-23" while the landing commit is dated 2026-08-24. + +### History foundation and the application boundary + +- **drop-panicking-progress-constructor** (Δ−12). `ApplicationProgress::new` + panics on an incoherent pair and has only test callers; its own doc says to + use `try_new` on the only path that constructs one from data. Delete it; + tests become `try_new(..).expect(..)`. Register finding 17's category. +- **defer-era-newtypes-to-track3** (Δ−110). The schema slice (`history_state` + columns, five triggers, the generation bump inside `cascade_and_reopen`) is + cheap to carry and expensive to retrofit, and should stay. The Rust surface + (`EraId` with Display/Debug/TryFrom, a three-variant parse error, + `RecoveryGeneration`, `HistoryVersion`) has zero production readers; the WS + feed destructures the coordinate away with `..` (`l2_tx_feed/mod.rs:299-315`), + and Track 3 says the era leg is explicitly unconfirmed by the consumer. + Alternative: represent the era as `[u8; 16]` at the storage boundary and let + Track 3 introduce the newtypes beside the wire codec. Low cost either way. +- **drop-uuid-version-variant-checks** (Δ−30). `mint_era_id` + (`open.rs:242-261`) stamps v4/RFC-4122 bits into a random blob, then the Rust + constructor and a SQL `CHECK` verify that self-imposed constant at three + points, for a token whose only semantics is equality. Keep the 16-byte + newtype, length check, and hyphenated Display; drop the version/variant + stamping and checks (+6 bits of entropy). The one argument for keeping it is + a future strict-UUID consumer, a Track 3 wire concern; if kept, reword the doc + from "must carry" to "presentational contract for the future wire form". +- **single-enforcement-for-mapping-contiguity** and + **drop-duplicate-offset-assert** (Δ−15). `attach_executed_inputs_in` + (`history.rs:116-133`) recomputes the exact predicate + `trg_executed_inputs_contiguous` (`0001_schema.sql:562-575`) enforces, on the + accepted user-op path with the latency contract — one `query_history_state` + read plus one `MAX(executed_input_offset)` probe per chunk. Its own comment + says the schema independently enforces the rule. Keep one enforcement point, + preferably the trigger (cannot be bypassed by any writer, aborts the + transaction rather than unwinding a panic through the lane). For directs the + offset is checked a third time by the derive-and-compare below. +- **drop-terminal-fault-typed-reader** (Δ−55). `latest_terminal_fault`, + `TerminalFault`, `LifecycleCommand::parse`, and the two `Malformed` variants + that only report a malformed black-box row exist to serve three test + assertions; an empty cause is already impossible at the engine + (`0001_schema.sql:628-630`). Contradicted in part by + `startup_log_last_terminal_fault` above, which would give the reader a + production caller; decide the black box's reader story once. +- **single-admission-implementation-for-setup** (Δ−25). + `preflight_lifecycle_command` has two callers (`run`, `flush`); setup and + rebuild go through `admit_setup_lifecycle` (`setup/mod.rs:361-388`), which + re-implements the same two facts with different semantics (an + already-complete plain setup is a no-op success there, `NotAdmissible` in + the lifecycle module). Make `preflight_lifecycle_command` return a three-way + admission for setup/rebuild and have setup call it. Also `run` calls the + preflight (which refuses without `setup_complete`) and then + `load_setup_identity` re-checks completion with a different error type; drop + the second check. + +### Lane and storage + +- **narrow-direct-attribution-cross-check** (Δ−8). `persist_frame_direct_sequence` + (`mutations.rs:168-181`) re-derives every direct's sender over the whole + drained range and asserts vector equality with the lane's receipts inside the + reconciliation commit; the lane read the same rows moments earlier. Cheaper + shapes: carry the skipped-submitter count and assert + `executions.len() + skipped == range.len()` plus first/last offset, or keep + the derive behind `cfg(debug_assertions)`. The honest answer depends on the + catch-up ACK measurement the register already owes (5,000 directs over a + 7,200-block jump in one turn). + +### Tests and e2e + +- **gate-remaining-test-only-storage-api** (Δ−40). `latest_batch_index` + (`l1_submission.rs:97`), `ordered_l2_txs_for_batch` (`:129`), and + `promote_finalized` (`snapshot_dumps.rs:182`) are `pub` with only test + callers; `promote_finalized` can promote without the lane's inclusion-block + and lease invariants. Delete the first two (fold into their tests), gate the + third. This is what the in-crate test move was supposed to unlock (register + finding 17). +- **drop-tryfrom-accepts-tests** (Δ−35). Five `*_accepts_*` tests in + `storage/convert.rs` assert that `std::convert::TryFrom` is correct; keep + every `should_panic` twin (they pin the settled decode policy) and + `prepare_time_sql_failures_classify_persistent_in_both_spellings`. Also + `era_id_displays_canonical_lowercase_hyphenated_form` pins a Display string no + consumer parses. Record in the register that the 21 new + `#[should_panic(expected = ..)]` attributes are accepted panic-message + coupling. +- **unify-harness-chain-clock** (Δ+60). Four notions of block time exist: + `SECONDS_PER_BLOCK = 12` duplicated at `rollups.rs:264` and + `sequencer.rs:876`, `LIVE_L1_BLOCK_INTERVAL_SECONDS = 1`, `BOOT_L1_MINE_INTERVAL + = 1 s`, and the sequencer's configured `seconds_per_block = 12`; e2e + correctness depends on their unwritten relationship staying under the 12 s + clock-usability threshold. `advance_live_frame_until_covers` + (`test_cases.rs:480-514`) can drive L1 roughly 20× ahead of the process + clock per iteration. Give the harness one `ChainClock` owned by the devnet + stack, constructed from the value passed to the sequencer, with + `advance(Duration)` and `mine_live(n)` deriving from it, and one post-mining + check `l1_head_timestamp − faketime_now < seconds_per_block` so drift fails + loudly in the harness. Two of the three re-staged scenarios are principled + and strictly stronger (`sequencer_outage_danger_zone_tip_cascade` now asserts + an invalidation; `wall_clock_backward_jump_retries_then_recovers` is the only + per-variant exit-code e2e in the suite). +- **replace-timewarp-tip-injection** (Δ−10). `aging_open_tip_runtime_danger_zone_exit_test` + injects a wedged lane by mining 1,150 blocks with wall time frozen (a chain + 3.8 h in the future), then compensates with `mine_live_l1_blocks(1)` plus an + absolute faketime offset, and greps the log to prove the future-dated view did + not route into the clock-fallback arm. Replace the injection with a + lane-level one (a `--freeze-frame-clock` test dial beside the existing + batch-open dial), advance wall and L1 together, and delete the compensation + and the log assertion. Minimum fix: assert exit code 10 instead of the log. + Also `set_faketime_offset` resets the cumulative counter while leaving an + absolute offset in the rc file, so a later `advance_wall_and_mine` in the + same scenario would regress the child clock. +- **replace-watchdog-sleep-assertions** (Δ−20). Two watchdog tests are + negative assertions implemented as `recv_timeout(250 ms).is_err()`; + unfalsifiable by slowness. Expose `is_watchdog_armed()` under `#[cfg(test)]` + or have the injected abort action record whether a deadline was scheduled. +- **merge-detector-arm-mapping-tests** (Δ−25). Three tests cover the + 13-line `FirstExit::detector`; merge into one table test over the four join + shapes. The composed containment tests the 2026-08-23 refutation protects are + untouched. +- **document_second_half_clock_assumptions** (Δ+12). Record at + `test_cases.rs:3168-3176` that the `+1` alignment margin is not the real + margin (Anvil block timestamps track wall time) and that the single + `mine_live_l1_blocks(1)` refresh has one block of headroom only because Anvil + runs with `--slots-in-an-epoch 1`. +- Also open: `RuntimeScope::default()` (`shutdown.rs:258-267`) leaks one temp + directory per construction via `mem::forget`; worth asking whether a + `(RuntimeScope, TempDir)` guard should be the only shape. + +### Documentation corpus + +- **refuted-evidence-grades** (Δ+10). Give each refuted entry an `evidence:` + line naming the file/line or measurement a reader can re-run; demote entries + that cannot produce one to "declined, no evidence recorded". Restore the + deleted cost datum to the per-chunk divergence-query entry (the + pre-distillation ADR read "every roughly 14-ms user-op chunk"; "14 ms" + appears in zero markdown files now). Name the select arm in the + homogeneous-list entry's title, since the `Vec<(WorkerId, ComponentShutdown)>` + shape now exists in the tree for cleanup. +- **collapse-six-stubs** (Δ−110, −6 files). Six of the eight dated ledgers are + 15–20 line stubs carrying a verdict plus a pointer; collapse them into a + "Review history" table at the bottom of the register. Keep the two August + ledgers (they carry the only re-verifiable evidence in the corpus). +- **adr-dedupe-vs-register-and-invariants** (208 → ~70 lines). Each ADR + mechanism is also described in the invariants check policy, AGENTS.md, the + recovery README, the threat model, the runbook, and module docs; five of six + rejected alternatives are also in the register's refuted list, and the two + point at each other circularly. Cut the ADR to context, the policy statement, + and four mechanism names with pointers; move the rejected-alternatives + arguments into the register so there is one home. +- **single-home-divergence-freeze** (Δ−60). `docs/invariants.md:353-372` and + `docs/recovery/README.md:398-415` are the same four sentences; the ADR's G3 + and `AGENTS.md:264` are third and fourth compressions. I15 owns the runtime + reaction and race bound; the others link. +- **agents-hotpath-to-pointers** (50 → ~20 lines). `AGENTS.md:255-304` + restates I2, I3, I9/I15, I17, I18, and the admission policy in fifteen + paragraph-length bullets, violating its own line-475 rule; the good pattern + is already used at `AGENTS.md:118-121` and `:324`. +- **module-docs-explain-not-defend** (Δ−15). Strike the four defensive + clauses (`workers.rs:21-27` "they are the enforcement, not style"; + `error.rs:10-13`'s dated `RunError` history and stale "acknowledge"; + `shutdown.rs:20-23`; `storage/recovery.rs:15-23`), and fix the three "journal" + usages. +- **finish-the-codename-sweep** (~14 one-line edits). Residue: `history.rs:202` + ("L2"), `l1_inputs.rs:41` ("H6"), `wallet.rs:101` ("D10") added by this + branch; `provider.rs:160,234`, `e2e_sequencer.rs:411,429,537`, + `tests/harness/src/sequencer.rs:33,38,302,305,583`, `test_cases.rs:3039,3924` + predate it. Track 6's requirement labels R1–R5 collide with the codename + map's R1–R5. +- **proportionality-measured**. Measured: ~8,026 lines of standalone doc/spec, + ~6,792 comment lines, ~21,200 lines of production Rust, roughly 0.7 prose + lines per code line; the branch's own margin is one doc line per six code + lines. Volume is defensible; the unstated fan-out is not. Either adopt a + single-home rule with a named canonical copy per mechanism, or write down + that redundancy is deliberate and name the canonical copy. +- Also open: `docs/plans/` is listed as timeless in AGENTS.md but the tracks + board is a dated status board; the deleted terminal-containment plan's + marker-file protocol has no refuted entry anywhere. + +### CI + +- **binary_disables_ansi_when_not_a_tty** (Δ+4, rank 1). Add + `.with_ansi(std::io::stdout().is_terminal())` to both wallet-sequencer mains + (`IsTerminal` is std). Independent of the test: a daemon writing to a pipe, + file, or journald must not emit SGR escapes. +- **assert_exit_class_not_log_text** (Δ−4, rank 2). Replace the log grep with + `exit.code() == Some(10)`; `TipInDanger` projects to 10 and the clock + fallback to 20. Caveat: 10 does not separate `TipInDanger` from + `ClosedBatchInDanger`; if that matters, keep one check anchored on the + never-styled substring `TipInDanger(` alone. +- **harness_pins_no_color** (Δ+2, rank 3). Pin `NO_COLOR=1` beside the + `RUST_LOG` pins at `sequencer.rs:1300` and `:1394` as hermeticity, not as the + fix. +- **strip_ansi_in_assertion** — rejected by its author; recorded so it is not + re-proposed. +- **centralize_tracing_init_in_run_main** (Δ−10, optional). The two mains are + byte-identical apart from the config constructor; `run_main` already owns the + exit-code contract and could own log rendering. A library installing a + global subscriber is a deliberate boundary decision, not part of the CI fix. + +### Roadmap items (plan, not simplification) + +- **pr-body-rewrite** — a ~270-word draft exists in the fleet output; the + title should name the actual scope and the body must carry the two breaking + changes (baseline migration rewritten in place; `Application` hooks renamed). +- **pre-draft-ci-fix**, **pre-draft-metadata** — the two blockers; plus + `docs/plans/2026-07-coordination-tracks.md`'s "ready for its PR against + main" line and the register's verification date. +- **fold-before-merge** — findings 4 (flusher `error!` → `warn!`), 5 (`/tx` + 500 body echoing `AppError` strings), 17 and the second half of 10 (gate the + test-only storage surface), and the false `debug_assert` comment at + `sequencer-core/src/fee.rs:228`. +- **followup-1-submitter** (findings 1–3, ~250 lines), **followup-2-schema** + (findings 9, 11, 18, while the baseline-rewrite window is open), + **followup-3-measure** (the 500 ms objective and catch-up ACK p99), + **followup-4-harness** (the owed levers and the e2es they unlock, split by + lever), **followup-5/6/7-track3** (typed history foundation and + `GET /history-version`; finalized replay routes, gated on consumer decisions + 2 and 3; the `/ws/subscribe` cutover absorbing findings 6 and 7), + **followup-8-track6** (working-image `Application` API; breaks the same trait + this PR breaks). + +## Recommended split + +**In this PR before it leaves draft:** the CI fix (binary ANSI plus the exit +code assertion), the PR title, body, and breaking-change notes, the two doc +lines that go false on merge, the prose-versus-types honesty sweep (token +coverage claim, "non-clone", the witness wording, the three "journal" words, +the "acknowledge" mention, the README type name, the two-row black-box shape), +the five mechanical register items, gating `ensure_open_tip`, and deleting the +panicking progress constructor. Each touches files the branch already rewrites +and none changes behaviour a reviewer has not already seen. + +**A focused successor PR:** the remaining confirmed items — `TipAlreadyOpen`, +the notification-half narrowing with its doc restatements, the kind-filtered +key-file error, the tip postcondition refuse, the exit-code table with the two +process-level assertions — plus whichever unverified runtime items survive +their own re-verification (the lease-release supervisor's home, the +non-optional inclusion block, the in-scope recorder). These change behaviour +or exit-code classification and deserve their own adversarial pass and tests. + +**Later, in order:** the doc single-home passes; submitter pacing; schema +hardening before first deployment; the latency measurement; the harness +levers; Track 3; Track 6. diff --git a/docs/review/register.md b/docs/review/register.md new file mode 100644 index 00000000..fbe33786 --- /dev/null +++ b/docs/review/register.md @@ -0,0 +1,491 @@ +# The Review Register + +The distilled outcome of every dated review ledger in this directory: what +is still **open**, what was **settled** (with its reasoning's current home), +and what was **refuted** and must not be re-proposed without new evidence. +Check the open section before touching related code; check the refuted +section before proposing a simplification or a new mechanism. The dated +ledgers beside this file are stubs preserving each review's scope and +verdict; process detail beyond that lives in git history. + +Statuses of findings 1–18 were verified against the tree on 2026-08-25. +Findings 19–31 and the 2026-09-03 refuted block come from the +[branch stock-take](2026-09-03-branch-stocktake.md), which records every +proposal that review raised, including the ones no jury examined. + +## Open findings + +Code findings, oldest first (file references are starting points, not exact +lines): + +1. **Submitter confirmation-timeout defeats pacing** — a watch timeout maps + to a successful `Submitted` tick, so the next tick immediately re-sends + the same payloads at the same nonces (usually "replacement underpriced") + before any sleep. `l1/submitter/poster.rs` + `worker.rs`; add a distinct + outcome that sleeps. +2. **A transient `SQLITE_BUSY` costs the submitter a full respawn** — 50 ms + reader `busy_timeout` plus every non-poster error ending the run. It now + classifies restartable rather than terminal, but the respawn+recovery + cost stands. Retry BUSY or use the writer-grade timeout for these reads. +3. **An undecodable own-sender payload stalls submission for the safe-lag** + — the poster hard-fails decode where both scheduler mirrors + skip-and-continue, so an operator's manual tx from the submitter EOA + wedges ticks until the block passes the safe head. Skip undecodable + own-sender payloads. +4. **Flusher logs `error!("flush retry: previous attempt timed out")` on + every healthy finality-wait pass** (`recovery/flusher.rs`). +5. **One `POST /tx` 500 path still echoes internals** — `AppError::Internal + { reason }` / `AppError::Io` strings pass verbatim into the 500 body + (`ingress/inclusion_lane/mod.rs` error mapping); the storage path is + already generic. +6. **WS session hygiene** — a mid-session transient read error tears down + with no close frame; a beyond-head `from_offset` idles forever (currently + e2e-pinned as intended — decide the contract, then re-pin). +7. **WS invalidation/rollback contract** — `/ws/subscribe` still pages by + physical rowid with no `HistoryVersion` claim, so a cursor-resumed + subscriber silently keeps invalidated rows across recovery. Interim + consumer rule: treat any socket drop as a potential discontinuity. + Closure is exclusively owned by the + [Track 3 handoff](../plans/2026-07-track3-feed-replay-design.md#7-ordered-implementation-handoff). +8. **Fee-determinism contract under-specified** — `fixed_mul`'s comment + claims a `debug_assert` that does not exist (high limbs silently drop), + and the LSB-first floor-after-each-multiply order is implemented but not + stated as contract (`sequencer-core/src/fee.rs`). Load-bearing for the + C++ scheduler port; interacts with the deferred fee-LUT track. +9. **`trg_enforce_nonce_contiguity` NULL hole** — a dangling parent makes + the comparison NULL and the trigger silent; mitigated by `foreign_keys=ON` + on every writer connection, but the trigger itself is not NULL-safe. +10. **`seal_and_open_next_batch` takes an unchecked `next_safe_block`** + (assert equality with the head or drop the parameter), and the bare + `close_frame_and_batch` remains ungated with test-only callers — + integration tests are in-crate now, so plain `#[cfg(test)]` suffices. +11. **Write-only columns** `safe_accepted_batches.{first_frame_safe_block, + inclusion_block}` have no production reader — drop or mark audit-only. +12. **`direct_q` is unbounded in the shared scheduler** — an adversarial + deposit flood is bounded in time (force-drain) but not bytes; a + per-input cap or byte budget closes a (very expensive) guest-OOM vector. +13. **`MAX_BATCH_METADATA_BYTES` (71) understates real SSZ per-op overhead** + (~83+ with offsets) — byte budgeting undercounts ~15% for max-payload + ops (`sequencer-core/src/user_op.rs`). +14. **Wallet snapshot decode accepts unsorted entries** while encode sorts — + enforce strictly-ascending addresses (subsumes the duplicate check) or + drop the canonical-decode pretense (`app-core/src/wallet_snapshot.rs`). +15. **Reader and submitter re-open `Storage` per tick** — a held connection + per worker drops per-tick overhead. Low priority. +16. **`should_retry_with_partition` substring-matches the Debug format** — + consciously accepted and regression-pinned against alloy's format; + revisit with structured JSON-RPC codes. +17. **Test-only surface still presenting as production API** — delete + `latest_batch_index` and `ordered_l2_txs_for_batch` (only their own + tests call them); gate `promote_finalized` behind `#[cfg(test)]` + (`safe_input_end_exclusive` has a live reader-path caller and stays). +18. **`frames` lacks the immutability triggers `batches` got** — `fee` and + `safe_block` are documented immutable but convention-protected only. +19. **A missing or unreadable key file exits 1 and restart-loops** — + `resolve_key_source` (`commands/config.rs`) returns a bare `io::Error` + that lands in `CommandError::Io`, while bad key *content* in the same file + exits 30. Kind-filter into a distinct `KeySourceUnreadable { path, kind }` + (NotFound/PermissionDenied/InvalidData/IsADirectory/NotADirectory + terminal; EIO and friends operational); never echo contents. + Jury-confirmed. +20. **`StaleDecision` cannot name the tip-present cause** — + `ensure_open_tip_for_recovery` raises `{ expected: Safe, actual: Safe }` + when the Tip already exists, and `recovery_tests.rs` pins that line. Add + `TipAlreadyOpen` plus a paired retry reason so `classify_mutation` carries + it to the operator. Jury-confirmed; production-unreachable. +21. **`Storage::ensure_open_tip` is `pub` with zero production callers** — a + new instance of 17 created by this branch, beside its guarded replacement. + Gate `#[cfg(test)] pub(crate)`; fix the two intra-doc links and + `docs/snapshots/lifecycle.md`'s genesis-Tip claim. Jury-confirmed. +22. **Detector and reader take `RuntimeScope` where `ShutdownSignal` + suffices** — each uses the scope only for `wait_for_shutdown` and already + holds a construction-required `ProcessLock`. Narrow, and restate the three + doc comments (`commands/run/workers.rs`, `runtime/shutdown.rs`) that claim + lock retention through scope clones. Jury-confirmed 2–1. +23. **`drive_recovery`'s one cycle has no enforced postcondition** — + `Repaired` + `Safe` + no Tip → `EnsureOpenTip` → `Repaired`, with no + watchdog on the boot path; `admission.tla` hardcodes `hasOpenTip' = TRUE`. + Return a typed `refuse` (exit 30 — not a `debug_assert`, not + `StaleDecision`) after `open_fresh_tip_in_tx` in the guarded phase, or make + the reducer arm a terminal `Refuse`. Jury-confirmed 2–1. +24. **A contained run writes two `terminal_faults` rows** — the in-scope + recorder's raw cause, then the bracket's prefixed cause; + `latest_terminal_fault` returns the second. Document the shape (recorder, + ADR, runbook); do not dedupe by variant — the recorder swallows its own + failures, so the bracket write is a genuine retry. Jury-confirmed. +25. **Prose claims more than the types enforce** — `authorize()`'s doc names + four compile-forced primitives (three are; the HTTP 200 publication gate in + `ingress/api.rs` and the two lane mutation commits are hand-placed + consults); `RecoveryProgress` is called "non-clone" in the recovery README, + `admission.tla`, and `recovery/mod.rs` but derives `Copy`; + `process_lock.rs`'s witness comment is narrower than the predicate; + "journal" survives in `storage/history.rs` and `commands/run/workers.rs`; + `commands/error.rs:11` names an "acknowledge" command; the recovery README + names a `DangerDetectorExit` type that does not exist. Verified first-hand. +26. **`finalized_state` handles an impossible `None`** — `LeasedDump` is + shared between the finalized and latest lease queries, so the `NOT NULL` + `inclusion_block` arrives as an `Option` and a `None` escalates to + containment (`egress/api/snapshot.rs`). Give the finalized lease its own + non-optional type and delete the branch. Verified first-hand; no jury. +27. **`RecoveryFailure::Provider(String)` carries two verdicts** — the + startup flush maps `ChainIdRpc` → retry and `Create` → refuse into one + variant, and nothing pins either arm; `classify_input_reader` has no doc + comment and its `Bootstrap`/`Join` refusals are unpinned; a pre-v3 + InputBox exits 1 under `setup` and 30 under `run`. Surfaced by the + refuters; split the variant and pin both polarities. +28. **Setup admission is implemented twice with different semantics** — + `preflight_lifecycle_command` (used by `run`/`flush`) and + `admit_setup_lifecycle` (`commands/setup/mod.rs`) each check the same two + facts; an already-complete plain setup is a no-op in one and + `NotAdmissible` in the other, and `load_setup_identity` re-checks + completion with a third error type. Unverified. +29. **`http.rs` hosts a runtime component** — the ~175-line supervised + lease-release queue with two containment sites belongs in the egress + snapshot module that owns leases; its queue is unbounded and capped only by + concurrent snapshot requests. Unverified; weight, not safety. +30. **Harness block-time notions are unreconciled** — four constants with an + unasserted relationship; `advance_live_frame_until_covers` can drive L1 + ~20× ahead of the process clock; `aging_open_tip_runtime_danger_zone_exit_test` + stages an impossible chain and greps the rendered log (the CI red). One + `ChainClock` authority plus a post-mining drift check; assert exit 10 + instead of the log. The log grep is verified first-hand; the rest is + unverified. +31. **`log_gas_price_updated_at_ms` is production-write-only** — written on + every refresh, read only by tests, yet the threat model cites it as the + telemetry that justifies having no expiry gate. Surface it (health field + or the transient-refresh warn) or drop it and fix the sentence. + Unverified. + +Open maintainer decisions: + +- **What the 500 ms acknowledgement contract is *for*** — it is 8× above the + worst measured value and shapes nothing today; either it encodes + catch-up-overlap headroom (then measure that) or restate the objective. +- **Catch-up ACK-latency measurement** owed to the benchmark harness: ACK + p99 *during* an epoch-sized catch-up reconciliation turn (the in-crate + seed test digests 5,000 directs over a 7,200-block jump in one turn). +- **Track 6 with Bart**: the hardlink-suitability dispute and the + changed-era bootstrap contract (see the tracks doc). + +## Owed tests + +- **Arm-ordering discriminating test**: both `ClosedBatchInDanger` and + `TipInDanger` genuinely in danger; assert Closed wins (today pinned only + incidentally by an equally-aged fixture). +- **Fail-loud halves**: no test references `CatchUpError::NoSnapshot` or + `InclusionLaneError::NoOpenTip`. +- **`EstimatedBatchInDanger` e2e** (recipe: mine ~800 blocks, faketime + +30 min without mining, respawn → refusal with zero invalidations). +- **Process-level divergence scenario** via `respawn_until_stable` (storage + and reducer coverage exists; the end-to-end freeze/refuse loop does not). +- **Per-variant exit-code e2e assertions** (failure-path e2es assert only + `!success()`) and a process-level SIGTERM→0 assertion. Assert integer + literals: the `EXIT_*` values appear only at their declarations, so + renumbering `EXIT_TERMINAL` passes the suite today. Cheapest 30-class case: + `run` on a never-set-up data directory. +- **Polarity pins for `classify_input_reader`'s `Bootstrap`/`Join` refusals**, + and a unit test of the production phase→progress mapping in + `ProductionRecoveryDriver::perform` (the scripted-driver traces exercise the + test double's copy of it). +- **Full-tear cascade on a recovered (anchor = `N'`) tree** re-rooting at + `N'` (anchor unit mechanics are covered; this end-to-end shape is not). +- **Uniswap-mode fee oracle end-to-end**: every fixture and e2e pins fixed + mode, so no Uniswap-mode sequencer boots in tests. Setup validation, + RPC-free runtime source construction, transient quote retention, and + terminal misconfiguration are source-boundary-pinned in-crate as of + 2026-08-25; a real E2E still needs a mock pool — decide whether that extra + harness is worth its weight. +- **True same-block direct-input ordering end-to-end**: the renamed + `multi_deposit_reconciliation_test` covers multiple accumulated directs, but + default Anvil automining puts its portal deposits in distinct blocks. A real + same-block test needs queued portal sends, one explicit mine, equal receipt + block assertions, and WS order/block attribution. +- **Verify-then-write-or-strike** (status uncertain on 2026-08-22): the + encoded-wire-frame stamp at an advanced safe head; the wallet + insufficient-balance silent no-op and replay-determinism pins; the + young-never-submitted-batch cascade-policy pin; the `recover_aging_tip` + torn/no-Tip entry; the cascade-with-backward-clock pin. +- **The batch-close failure half of I7**: pre-insert a `dumps` row with a + colliding prefix so the seal transaction fails on UNIQUE, and assert the + batch stays the open Tip. Companion state variant: delete the directory + under a DB-referenced snapshot row and assert the loud terminal shape + (the WAL-rewind *cause* stays unsimulable). +- **Harness levers to build with their tests**: pending-tx capture + + re-inject (`txpool_content`/raw-tx before `drop_all_pending_txs`, then + `eth_sendRawTransaction`) → unlocks the zombie e2e, the headline + adversarial scenario; snapshot lifecycle observability (DB readers for + the snapshot tables + dump-dir inspection) → the take/promote/GC/lease + e2e, plus finally *asserting* warm-resume-from-dump (every restart test + exercises it, none asserts it — a silent fall-back to genesis replay + would pass everything, just slower); a bare second Anvil + (`--chain-id `) + mid-run endpoint override → the wrong-chain + e2e; kill-at-log-marker → the flush-completion/cascade-commit crash + window; SQLITE_BUSY injection → the submitter/WS BUSY items. + **Recorded do-not-build**: a split-view/response-rewriting L7 proxy + (unit-level provider mocks instead; e2e validates only the passing + path) and fsync/power-loss WAL-rewind injection (out of scope — + state-construction variants cover the detectable halves). + +## Settled decisions + +Each entry: the decision, its reason, and where the reasoning now lives. + +- **Write-before-broadcast watermark** (2026-06): the flush's completion + anchor is durable, not the local pool's memory → I14. +- **Content-identity check, gated on full acceptance** (2026-06): accepted + landings compare by content hash; content-equal copies are effect-equal, + so no batch identifier is needed; detection freezes the frontier + atomically with the detecting sync → I9, I15. +- **`synchronous=FULL`** (2026-06): externalization rides on commits, so + every commit fsyncs; noise-level cost on NVMe → `storage/open.rs` doc. +- **Cockroach recovery's flush is best-effort by construction** (2026-06): + the wiped DB destroys the watermark, so the flush resolves only what the + provider remembers, and plain `setup`'s detection gate shares the same + false negative. Accepted because the content-identity check turns the + residual zombie from silent divergence into a detected freeze (repair: + wipe and re-run). Recorded option if ever needed: an operator-supplied + flush floor from the old DB's watermark — fail-safe under corruption + (too high wastes no-ops; too low degrades to exactly best-effort) → + `cockroach.md` step 2. +- **Exit-code contract** (2026-06, panics-terminal amendment 2026-07): the + orchestrator must not parse logs; 10/20/30/40 by restart productivity → + `commands/error.rs` + the operator runbook. +- **Fail-loud check policy replaces "no defense-in-depth"** (2026-06): the + line is loud-vs-silent, not self-doubt; assertions must check real + invariants (the wall-clock CHECK cautionary tale) → the invariants check + policy. +- **Scoped pending clear** (2026-06): delete only pending rows at/above the + cascade pivot, in the cascade's transaction → I5. +- **Batch-tree anchor, not a sealed sentinel** (2026-06-25): the parentless + root carries the anchor nonce, exact-matched by the contiguity trigger → + I16. +- **`N` is trusted; no recovery-time verifier** (2026-06-26): a + sequencer-produced finalized dump cannot carry a wrong `N` by + construction; only wrong-low is caught at `run` → `cockroach.md` data + dictionary. +- **Anchor-aware frontier; recovery defers population** (2026-06-26): + below-anchor landings are trusted collapsed history → I15. +- **Recovery drain caps at `C`** (2026-06): `(C, H1]` deposits stay + undrained so `run` leads them exactly once → `cockroach.md` steps 3/6. +- **Don't resurrect TEST_PLAN.md** (2026-06): the scenario matrix rotted + once; owed tests live here as a dated, finite list. +- **The authority boundary** (2026-08, re-evaluated 2026-08-02): four + mechanisms — `RuntimeScope`, fact-derived admission, the pure recovery + reducer, SQLite-centered runtime with the two-regime lane → the + [ADR](../plans/2026-08-authority-boundary-adr.md). +- **Storage decode policy** (2026-07): fail-loud for contract-impossible + values; the named `saturating_query_bound` only where clamping preserves + the predicate → `storage/convert.rs` + the check policy. +- **The calibration rule** (2026-08-18): the complexity budget belongs to + concurrency, durability, and hostile-L1 robustness → AGENTS.md design + principles. +- **The `Authorized` externalization token** (2026-08-18): the containment + consult is a compile-time obligation of the effect functions → + `runtime/shutdown.rs`, ADR mechanism 1. +- **Module homing** (2026-08-19): command brackets in `commands/` (with + config + the `CommandError` taxonomy), the capability substrate alone in + `runtime/`, `L1Config` in `l1/`; a full merge was refused because the + substrate is consumed crate-wide. `http.rs` stays whole until the + ingress/egress listener split forces it apart. +- **Lifecycle: facts govern; the black box records** (L1 2026-08-18 → + L2 2026-08-19 → L3 2026-08-22): admission is three facts; no state + machine, no acknowledgement (it carried no machine-consumed decision); + telemetry writes are verdict-neutral; terminal faults refuse at + re-detection with the residual recorded in the threat model → the ADR, + the invariants check policy, and the two dated ledgers. +- **Misconfig-poison taxonomy** (opened 2026-08-18, closed by L3): there is + no poison; misconfig is terminal by exit code only, and a fixed config + boots cleanly. +- **Submitter-key redaction** (2026-08-24): the key enters the process as + `SubmitterKey` at the clap edge — `Debug` redacts, no `Display` exists, + and the raw hex is reachable only through `expose_secret`, so every + consumer of the secret is greppable. The key's public identity is the + pinned `batch_submitter_address` beside it. Closes the former + Debug-derive open finding → `l1/mod.rs`. Deferred separately: the startup + log prints the full RPC URL, which the help-leak test treats as + token-bearing. + +## Refuted — do not re-propose without new evidence + +From the 2026-06 reviews: + +- **`scheduler_accepts` omitting the two structural rejections is a bug** — + deliberate self-trust; the simulator runs only over our own well-formed + batches; the worst case is covered by the content-identity check + (documented in `scheduler-semantics.md`, duality-test-pinned). +- **Sealed `N'-1` sentinel batch** for recovery rooting — a valid closed + sentinel is a legal cascade pivot; nothing stops a runtime cascade from + invalidating it, after which recovery ABORTs. Its safety rested on an + unenforced assumption. +- **Recovery-time `N` cross-check** — circular: every cheap recomputation + seeds from the `N` it would check; the only independent check is a + from-genesis L1 replay, deliberately not built. +- **`FoldInputSource` abstraction** — a wrapper over a single call site; + revisit only if a second fold-input source appears. +- Wire-fee-exponent panics, stalled-WS DoS, operator `WalletConfig` dead on + warm start, snapshot bytes history-dependent — each verified as + deliberate/out-of-scope (see the threat model's scoping). + +From the ADR re-evaluation (2026-08-01/02): + +- **`RunEpoch`**, **`EffectGate`**, **`LiveKernel`** — see the ADR's + rejected-alternatives section for each argument. +- **A generic command controller** over setup/rebuild/run/maintenance — + unrelated facts; a larger state machine closing no hole. +- **A per-chunk divergence query / reader mailbox** on the hot path — the + check is not a divergence oracle; the cost buys no complete boundary. +From the 2026-08-18 adversarial pass: + +- **Merging `Workers::finish`'s two drain modes by re-awaiting the primary** + — the winning select arm consumed the handle's completion (the select + borrows `&mut self.server` etc., so the handle is still in the cleanup + set); re-polling it panics ("JoinHandle polled after completion"), + unwinding through `ShutdownOnDrop` into containment — a benign stop + becomes a poisoned data directory. Scope-narrowed 2026-08-23: the 2026-08 + source read "as sketched" / "the naive merge"; distillation dropped the + qualifier. A merge that removes the primary from the cleanup set before + draining — keeping the `expect` that it was present — is settled, not + refuted (landed as `finish`'s one-loop/two-phase shape). +- **Removing the post-commit accessor-coherence assertion** — it is the + only guard in the two contexts with no database backstop (canonical + RISC-V fold, `fold_replay`). +- **"The three-variant frame-drain writer family is bloat"** — backwards: + the raw physical writers are `#[cfg(test)]`-demoted; production has one + way to write a frame. +- **`FuturesUnordered` for cleanup polling** — not worth promoting a + dev-only dependency tree to delete one small hand-written future. + +From the 2026-08-23 run-glue simplification pass: + +- **`Workers` as a homogeneous component list** (a `Vec<(WorkerId, + ComponentShutdown)>` built at launch, one select arm racing the list) — + it makes the "no `.await` between `Poll::Ready` and `swap_remove`" + property `select!`-load-bearing and untested (violation loses a worker + exit *and* panics on re-poll); converts the asserted primary-in-set + precondition into an unchecked cross-function assumption whose failure is + the benign-stop-to-exit-30 outcome; deletes the composed detector + select-mapping tests; and makes a zero-component race representable. The + real hole it targeted (select arms were the one per-worker site not + compile-forced) is closed by the exhaustive `let Self { .. }` destructure + in `select_first_exit` instead. +- **`UniswapConfig::pinned(&identity)`** — the exhaustive + `FeeOracleIdentity` match in the run bracket IS the launch decision for + the optional oracle worker; moving it behind an `Option` + constructor makes a future identity variant compile while silently + launching no worker, and the chain-id-pairing guarantee it claimed is + already structural now that the identity travels whole inside `L1Config`. + +From the L3 review (2026-08-22): + +- **A durable boot gate on terminal verdicts** (a gate on a *verdict*, i.e. + a non-fact) — it needs an acknowledgement to exit, and the acknowledgement + carries no information the fact-derived reducer doesn't re-derive. A + verdict-neutral startup *read* of the black box is not covered by this + entry. +- **A boot-time full-integrity sweep** — expensive machinery that still + cannot catch semantic violations outside its read set; the residual + window is recorded and bounded instead. +- **A boot assert on `finalized_snapshot.inclusion_block`** — the column is + `NOT NULL` at the engine; the claimed gap does not exist. (The runtime + `Option` branch on the same column is finding 26.) + +From the 2026-08-25 fee-oracle lifecycle pass (143a290; a single-pass +decision, not an adversarial review): + +- **Fee-price age as a runtime lifecycle gate** — setup owns the required + live quote; run starts from the persisted price and refreshes it best-effort. + Shared-endpoint staleness is already detected from safe-head progress, while + a pool-only outage is an explicitly accepted economic residual. Reintroduce + an expiry gate only with an independently derived economic bound and action, + not by borrowing the L1 liveness threshold. The telemetry this entry leans + on is production-write-only (finding 31). + +From the 2026-09-03 branch stock-take (three refuters per proposal; full +reasoning in the [ledger](2026-09-03-branch-stocktake.md)): + +- **`Workers` on a `JoinSet` fed by the shutdown waiters** — `from_select` + and `from_shutdown` deliberately read a worker's clean `Ok(())` two ways + (live: stopped unexpectedly; drain: graceful); one set collapses them and a + silently dead lane becomes exit 0. Evidence: `commands/error.rs:606-627`, + `commands/run/workers.rs:339-390`. The `swap_remove` hazard is real; a + cleanup-only set built inside `finish`, with the live race untouched, was + not examined. +- **Collapsing `PreparedRuntime` into an `async fn boot`** — + `fn launch(self, RuntimeAdmission) -> Workers` is non-async and non-`Result`, + so `?` and `.await` after admission are compile errors today; `boot` makes + them legal and the witness degenerates. Evidence: `workers.rs:288`, + `recovery/mod.rs:477-483`; 02a2b34 stopped here deliberately. +- **Making the `Authorized` token "uniform" by demoting the `/tx` 200 gate + to a bool** — the 200 body is the acknowledgement leaving the process; + `LeasedDumpBody` does not exist and `finalized_inclusion_block` has no + streaming primitive. Evidence: `ingress/api.rs:84-92`, + `egress/api/snapshot.rs:101-120,214`. The doc tightening survives (finding + 25). +- **Deleting the `#[from]` impls so a refusal reason cannot be typed into a + retry** — the enums are public with public variants; the longer spelling + still compiles and is the dominant idiom at all 15 sites. Evidence: + `recovery/mod.rs:45-49,109-115`. +- **One `is_terminal()` on `VerifiedSignerProviderError` read by both + tables** — the `From` impl performs no classification; the verdict is taken + later over `BootstrapError`, whose variants have four other producers. + Evidence: `commands/error.rs:671-683,216-260`; `l1/reader.rs:96-104` + already documents the phase pair. +- **Flattening `RecoveryError` into one enum with `is_retryable()`** — + `RecoveryFailure::Provider(String)` carries two verdicts, so a total + function over the value does not exist. Evidence: `recovery/mod.rs:429-438`. + Splitting the variant is a separate, sound fix (finding 27). +- **Moving the phase→progress mapping into `drive_recovery`** — `(Flush, + Done)` has no target without the observed block; this is + `PhaseCompletion`/`transition_after_phase`, deleted by ed41f9b. Deleting + `RecoveryDriver::admitted` survives. +- **Merging `FeeOracleMisconfig`/`FeeOracleFatal` and + `ChainIdRpc`/`DetectionNonceRead`** — bare-string producers at + `commands/setup/mod.rs:144,171` would misattribute an operator mistake in + the black box; the second merge demotes a compile-forced classification to a + string discriminant (refuted twice on 2026-08-23). +- **A `TerminalityOf` trait for `WorkerStop`** — admits the same wrong + classification as `|_| false`, installs a crate-wide answer for `io::Error` + that `dump_info.rs:49-58` contradicts, trips `private_bounds` under + `-D warnings`, and splits an inherent convention across eight types. +- **Wiring `check-admission` into CI as one line, and pruning three + "duplicate" settle actions** — no Nix or `.envrc` is tracked and `just` is + absent from the `rust` job; TLC checks the spec against itself; the actions + are state-neutral but `InspectRetry` encodes a recorded commitment. A + properly pinned standalone `formal` job survives as a proposal. +- **A sequencer-owned `AppWithProgress` wrapper replacing the progress + capabilities** — the pair lives inside the canonical SSZ bytes the watchdog + byte-compares and the canonical machine advances it inside its own + transition; cockroach recovery reads the clock from a dump into a wiped + database. Evidence: `examples/app-core/src/wallet_snapshot.rs:41-42`, + `commands/setup/mod.rs:433-451`. Record the composition in the application + contract (jury-confirmed as a do-not-adopt). + +Also standing, from the same reviews: the **do-not-simplify list** now +lives beside the invariants it protects +([`docs/invariants.md`](../invariants.md), "Do-not-simplify"), and these +deliberate declines keep their reasons — egress single-poller fan-out (no +need at current subscriber counts), `LeaseGuard` shared release channel, +`finalized_state` ETag on `l2_tx_index` (no reachable collision), +stale-skip on-chain report (scheduler protocol change; queue behind the +scheduler library), `Storage::read` commit-vs-rollback (no behavioral +difference). + +## Historical codename map + +Older commit messages and the pre-distillation ledgers (in git history) use +these codes; their concepts now live here: + +| Code | Concept | Current home | +|---|---|---| +| R1a | write-before-broadcast watermark | I14 | +| R1b | cockroach recovery's best-effort flush | `cockroach.md` step 2 + settled above | +| R2 | content-identity check | I9, I15 | +| R3 | `synchronous=FULL` decision | `storage/open.rs` | +| R4 | exit-code contract | `commands/error.rs`, runbook | +| R5 | fail-loud check policy | invariants check policy | +| F1–F10 | 2026-06 correctness findings | settled above; F7 = the open "WS invalidation/rollback contract" finding | +| I1–I20 | invariants (stable, still in use) | `docs/invariants.md` | +| D1–D11, H1–H14, S-A, P1–P8 | 2026-08-18 defects / harvest / structural fix / premise items | settled above + ADR | +| WP1–WP11 | 2026-06 work packages (all landed) | settled above | +| L1, L2, L3 | the lifecycle decisions (not Layer 1/2) | ADR mechanism 2 + the two August ledgers | +| S1–S7, A1–A12, B1–B5 | 2026-06 simplification queue / owed tests | open remnants above | diff --git a/docs/snapshots/lifecycle.md b/docs/snapshots/lifecycle.md index 27c92617..8d52b76d 100644 --- a/docs/snapshots/lifecycle.md +++ b/docs/snapshots/lifecycle.md @@ -30,14 +30,21 @@ aren't. Section references point to the full reasoning below. loads the resulting head from storage (fail-loud if absent), so it only ever *loads* — it never branches on tip existence or initializes one. (§7) - **A committed promotion implies an advanced drain.** Promotion is folded into - the drain's transaction (`close_frame_only_promoting`), so the two commit - together — this is what makes a crash safe. (§5, §6) + the drain's attributed transaction + (`close_frame_only_promoting_with_executions`), so promotion, physical + drain, and logical mappings commit together—this is what makes a crash safe. + (§5, §6) - **No dangling row.** No `dumps` row references a missing directory: create the file before the row; delete the row before the file. (§7) -- **One resume checkpoint.** The same row supplies both the `from_dump` prefix - and the replay offset, so loaded state and replay cursor can't drift. (§4) +- **One resume checkpoint.** The same row supplies the `from_dump` prefix, + physical replay cursor, and canonical executed-input count. Startup checks + the loaded app count before replay, so state and either coordinate cannot + drift. (§4) - **Snapshot `l2_tx_index` is the global valid replay head**, not the batch's own last offset — so an empty batch doesn't reset catch-up to genesis. (§3) +- **Snapshot `executed_input_count` is storage-derived `H`.** Registration + fails loud if the application's count differs from the canonical mapping; + promotion carries the count with the physical cursor. (§3–5) - **Storage is SQLite-only; the lane owns FS cleanup.** That boundary *is* the GC crash-ordering guarantee — don't push filesystem work into `storage/snapshot_dumps.rs`. (§7) @@ -49,7 +56,8 @@ aren't. Section references point to the full reasoning below. skipped checkpoints were never observable. Not a bug. (§5) - **`Storage::promote_finalized` (standalone) is `pub`, but production must not call it.** Promoting outside the drain transaction re-opens the wedge (§6); it - exists only for test setup. Production promotes via `close_frame_only_promoting`. + exists only for test setup. Production promotes via + `close_frame_only_promoting_with_executions`. - **Several snapshot `Storage` methods are `#[cfg(test)]`** — non-atomic siblings of the atomic production methods (`gc_dump_rows` vs `gc_unreferenced_dumps`; `acquire_dump_lease` vs `acquire_*_lease`; @@ -68,7 +76,8 @@ aren't. Section references point to the full reasoning below. ## 1. Purpose & model A snapshot is a durable copy of the application's canonical state at a known -point in the L2-tx stream. It exists for three consumers: +physical replay cursor and canonical executed-input boundary. It exists for +three consumers: - **Catch-up** (lane startup): instead of replaying the entire L2-tx history, the lane loads the freshest snapshot and replays only the tail after it — a @@ -77,15 +86,17 @@ point in the L2-tx stream. It exists for three consumers: sequencer's state against an independent canonical machine advanced through L1. - **Indexers** (operator): fetch the **latest** snapshot, then subscribe to the - L2-tx feed from that snapshot's offset. + L2-tx feed from that snapshot's offset. The current API exposes the physical + `l2_tx_index`; Track 3 will expose/admit the canonical + `executed_input_count` with `HistoryVersion`. Three SQLite tables back it (`storage/migrations/0001_schema.sql`): | Table | Holds | |----------------------|----------------------------------------------------| | `dumps` | `(id, prefix, lease_count)` — one row per on-disk dump directory | -| `pending_snapshots` | `(nonce, dump_id, l2_tx_index)` — snapshots of closed-but-not-yet-L1-confirmed batches | -| `finalized_snapshot` | single row `(dump_id, inclusion_block, l2_tx_index)` — the latest L1-confirmed state | +| `pending_snapshots` | `(nonce, dump_id, l2_tx_index, executed_input_count)` — snapshots of closed-but-not-yet-L1-confirmed batches | +| `finalized_snapshot` | single row `(dump_id, inclusion_block, l2_tx_index, executed_input_count)` — the latest L1-confirmed state | `prefix` is the **dump directory** — a structured dir the sequencer owns (`ingress/inclusion_lane/dump_info.rs`): @@ -109,6 +120,11 @@ closing the commit-then-stamp crash window). An in-place update of a file and GC — all keyed on the immutable directory path — are untouched. The dir name itself stays opaque; metadata lives only in `info.toml`. +`executed_input_count` is intentionally not another `info.toml` field. It is +already canonical application state inside `state`, while SQLite stores the +independent expected value used to reject a mismatched dump at startup. The +physical replay cursor remains sequencer-owned checkpoint metadata. + The split between **pending** and **finalized** mirrors the sequencer's optimism: a batch closes off-chain (soft) → its snapshot is *pending*; the batch lands safe on L1 → its snapshot is *promoted* to finalized. @@ -121,12 +137,19 @@ the GC crash-ordering (§7). ## 2. The always-load invariant **A finalized snapshot always exists by the time the lane starts.** The runtime -guarantees it in `Workers::spawn` (`runtime/workers.rs`): on cold start -`ensure_finalized_snapshot` consumes the genesis `Application`, writes it as a -dump, and registers it directly as finalized (bypassing pending); on warm start -it is a no-op. This gives catch-up a single unconditional path — there is always -*something* to load — and turns "no snapshot" into a violated invariant surfaced -fail-loud as `CatchUpError::NoSnapshot`, never a branch the happy path handles. +establishes it across the setup/run boundary: `setup` writes and registers the +genesis dump directly as finalized (bypassing pending) before atomically +committing setup completion. On every `run`, the startup reducer refuses a +missing finalized-snapshot fact before any provider call, and task-free +`PreparedRuntime::prepare` requires and re-stamps the referenced artifact before +durable runtime admission. This gives catch-up a single unconditional path — +there is always *something* to load — and turns "no snapshot" into a violated +invariant surfaced fail-loud as `CatchUpError::NoSnapshot`, never a branch the +happy path handles. +The same applies when the durable row exists but its referenced metadata or app +artifact is missing or structurally corrupt: startup classifies that provenance +as terminal instead of restart-looping. Other filesystem availability errors +remain operational. ## 3. Taking a snapshot at batch close @@ -155,36 +178,51 @@ the batch's own last offset. An empty batch (no sequenced txs of its own) thus inherits the prior head rather than recording genesis — otherwise catch-up from its promoted snapshot would replay the whole stream and double-apply it. +The same row records `executed_input_count` = storage-derived live head `H`. +The lane passes the count embedded in the just-dumped application; +`insert_pending_dump_in` asserts it equals the maximum current canonical +execution attribution (or era base `K`). This check is inside the +seal/open/snapshot transaction, so a disagreement cannot produce either a +sealed batch or a registered checkpoint. + ## 4. The resume checkpoint On startup the lane selects **one** checkpoint (`catch_up_snapshot`, in `catch_up.rs`): the latest pending snapshot if any, else finalized. The *same* -row supplies both `A::from_dump(&prefix)` and the catch-up replay offset -(`l2_tx_index`), so the loaded state and the replay cursor can never drift apart -(they come from one row). Loading from a *pending* (not-yet-L1-confirmed) +row supplies `A::from_dump(&prefix)`, physical catch-up cursor +`l2_tx_index`, and canonical `executed_input_count`. Before replay, startup +requires the loaded application's count to equal the stored count. During +replay, each executable physical row must carry exactly the app's current +count, while our batch-envelope rows must carry no mapping. These checks happen +before executing the row and make missing, extra, or wrong attribution a +terminal invariant failure rather than a repair/backfill path. Loading from a +*pending* (not-yet-L1-confirmed) snapshot is safe because danger-zone recovery clears any cascade-doomed pending **before** the lane starts (§8) — a surviving pending is either gold or legitimately in-flight under the optimistic model. ## 5. Promotion -As the lane advances the safe frontier (`maybe_advance_safe_frontier`), it walks -the newly-safe inputs. For each input that is one of *our* batches landing on -L1, `accepted_batch_nonce_at` (reading `safe_accepted_batches`, the +When the lane's five-safe-block clock criterion admits an L1-reconciliation +turn (`maybe_advance_safe_frontier`), it walks the complete accumulated +newly-safe range. For each input that is one of *our* batches landing on L1, +`accepted_batch_nonce_at` (reading `safe_accepted_batches`, the scheduler-acceptance view) yields its nonce. A `BlockObservation` accumulates the **highest accepted nonce seen in the range and the L1 block it landed in**. At range close the lane promotes that one `(nonce, block)` target. `promote_finalized` points the singleton `finalized_snapshot` at the pending -dump for `max_nonce`, carries over its `l2_tx_index`, and **deletes every +dump for `max_nonce`, carries over its `l2_tx_index` and +`executed_input_count`, and **deletes every pending row with `nonce <= max_nonce`** — the promoted one plus any stale rows behind it. ### Per-range, not per-block -Promotion happens **once per safe-frontier advance**, even when the range spans -several L1 blocks with several of our batches. This is sound, and loses nothing, -because of two facts: +Promotion happens **once per eligible clock/reconciliation turn**, even when +the range spans several L1 blocks with several of our batches. Safe-head +observations below the five-block threshold accumulate without draining or +promotion. This is sound, and loses nothing, because of two facts: - **Monotonic landing order.** L1 wallet nonces guarantee a higher nonce lands in a later-or-equal block, so the range's max nonce sits in its *latest* @@ -195,24 +233,29 @@ because of two facts: - **The intermediate checkpoints were never observable.** `finalized` is a single row the watchdog polls *asynchronously* — even with per-block promotion it can miss intermediates between polls. So "visits every block" was - never a guarantee; per-range removes a cadence nicety, not a contract. In - steady state a range is ~1 block anyway; the difference only appears during - multi-block catch-up, where a finalized that jumps to the latest block is - exactly what you want. - -`BlockObservation` (`snapshot.rs`) is therefore a **constant-memory -accumulator**: one `Option<(nonce, block)>`, `observe()` infallible and -storage-free, `promotion()` returning the target. It does no I/O on the hot -loop. + never a guarantee; per-range removes a cadence nicety, not a contract. The + five-block clock intentionally makes multi-block ranges normal, and a delayed + or epoch-sized safe-head jump may make them larger. Finalized state advances + directly to the latest accepted landing in the range; no intermediate + checkpoint is synthesized. + +`BlockObservation` (`snapshot.rs`) keeps one `Option<(nonce, block)>` for +promotion and the direct-execution receipts for the complete reconciliation +range. That vector is required to attach each canonical offset in the eventual +atomic frame transaction. It is confined to the deliberately slow L1 regime; +the user-op hot path does not use it, and scratch paging may bound input reads +without turning the logical reconciliation turn into resumable state. ### Atomic with the drain The promotion is **folded into the same transaction that advances the drain**: -`maybe_advance_safe_frontier` calls `close_frame_only_promoting`, which sequences -the drained safe inputs, rotates the frame, *and* runs `promote_finalized_in` — -all in one `write`. A crash therefore leaves promote + delete-pending + -drain-sequence either all committed or all rolled back. This is the fix for the -wedge in §6; see there for why a *separate* promotion is dangerous. +`maybe_advance_safe_frontier` calls +`close_frame_only_promoting_with_executions`, which sequences the drained safe +inputs, attaches their canonical execution offsets, rotates the frame, and +runs `promote_finalized_in`—all in one `write`. A crash therefore leaves +promote + delete-pending + drain-sequence + attribution either all committed +or all rolled back. This is the fix for the wedge in §6; see there for why a +*separate* promotion is dangerous. The standalone `Storage::promote_finalized` is retained only for test setup (it's the only way to *supersede* an existing finalized row, which @@ -286,10 +329,11 @@ such garbage (the superseded finalized, lower-nonce pendings). ### When GC runs -**After a promoting safe-frontier advance, on the lane's own thread** -(`maybe_advance_safe_frontier`, right after `close_frame_only_promoting` +**After a promoting clock/reconciliation turn, on the lane's own thread** +(`maybe_advance_safe_frontier`, right after +`close_frame_only_promoting_with_executions` commits — `run_gc::` when a promotion occurred). One full -`gc_unreferenced_dumps` pass per advance that promoted; it reclaims the +`gc_unreferenced_dumps` pass per turn that promoted; it reclaims the just-superseded finalized plus any earlier lease-released garbage. Why this, and not the alternatives: @@ -338,21 +382,34 @@ serializes against any concurrent writer. The streaming endpoints (`/finalized_state`, `/latest_snapshot`) must not have their dump GC'd mid-response. The lease read and the row read are **one atomic tx** (`acquire_finalized_lease` / `acquire_latest_snapshot_lease`), and the -handler holds the lease for the response lifetime via a **drop-guard** inside the -streaming body — so it releases on completion, error, *and* client disconnect. -Release is offloaded to `spawn_blocking` so the write-lock-contended release -never stalls an async worker. `reset_dump_leases` at startup is the crash -backstop. (Endpoint shapes: [`AGENTS.md`](../../AGENTS.md) and the root -[`README.md`](../../README.md).) +release guard is armed only after that transaction commits — a failed commit +cannot schedule a decrement for an increment that rolled back. The handler then +holds the lease for the response lifetime via the **drop-guard** inside the +streaming body, so it releases on completion, error, *and* client disconnect. +Releases are enqueued to a **supervised** blocking task set +(`http.rs::supervise_snapshot_releases`) that the HTTP worker drains before +exit classification, so no release can outlive the runtime's verdict. A +release failure is classified like any storage failure: a *persistent* error +(e.g. the lease row is gone — `StatementChangedRows` — or a persistent +open/migration failure) is a storage-invariant violation and takes the +runtime down terminally (exit 30); transient failures (BUSY, I/O) are logged +and left to the startup backstop. `reset_dump_leases` at startup remains the +crash backstop for releases that never ran. (Endpoint shapes: +[`AGENTS.md`](../../AGENTS.md) and the root [`README.md`](../../README.md).) ### Startup sequence -`Workers::spawn` runs five steps, order-critical: (1) `reset_dump_leases` -(clear stale leases from a crashed run), (2) `ensure_finalized_snapshot` -(genesis snapshot if cold), (3) `ensure_open_tip` (genesis Tip if cold — the -tip-existence invariant below; the lane loads the head itself after catch-up), -(4) `snapshot_gc_at_startup`, (5) `sweep_orphan_dumps` (remove on-disk dirs not -in `dumps`; runs *after* (2) so the genesis prefix is registered and not swept). +Before this sequence, the startup reducer has already required a finalized +snapshot fact and established a Tip through either guarded `EnsureOpenTip` or +an atomic recovery reopen. `PreparedRuntime::prepare` then calls +`startup_hygiene::run_snapshot_hygiene`, which runs five order-critical +steps before runtime admission, while no task +exists: (1) `reset_dump_leases` (clear stale leases from a crashed run), +(2) `require_finalized_snapshot`, (3) `restamp_finalized_promotion`, +(4) `snapshot_gc_at_startup`, and (5) `sweep_orphan_dumps` (remove on-disk dirs +not in `dumps`; the finalized prefix is already registered and cannot be +swept). Durable admission and the non-yielding worker launch follow only after +preparation completes and the reducer re-inspects current facts. ## 8. Recovery interaction @@ -361,8 +418,15 @@ Danger-zone recovery (`storage/recovery.rs`, see that the canonical stream will never reach. In the same transaction as the cascade it clears `pending_snapshots` **scoped to the cascade**: only rows with `nonce >= pivot.nonce` — exactly the cascaded batches' pendings, which -catch-up must never load. (Review F9; implemented in `cascade_and_reopen`, -the shared tail of both recovery paths.) +catch-up must never load (`cascade_and_reopen`, the shared tail of both +recovery paths). + +The same cascade retains the physical `sequenced_l2_txs` audit rows but deletes +their derived `executed_inputs` mappings, advances `RecoveryGeneration` once, +and opens the replacement Tip atomically. The surviving snapshot count is the +retained logical head; replacement history reuses the rewound suffix offsets +under the new generation. A crash cannot expose a new generation with old +mappings, or a rewound projection with doomed pending state. Pendings of *gold but not-yet-promoted* batches (landed and accepted while the process was down) carry lower nonces and **survive**: catch-up resumes @@ -372,8 +436,8 @@ promote-wedge **unrepresentable** rather than unreachable: any nonce the lane can later observe as accepted either has its pending row intact or belongs to a post-recovery batch with a fresh row. (The earlier blanket clear was safe only through a chain of cross-file couplings — same-tx full-backlog -reopen drain, `check_danger` arm ordering, frame-safe-block monotonicity — -documented in the 2026-06-10 review, F9.) In the `RecoverTip` path the +reopen drain, `check_danger` arm ordering, frame-safe-block +monotonicity.) In the `RecoverTip` path the scope deletes nothing: the Tip never has a pending row. `finalized` is untouched (its bytes are for an L1-confirmed batch, which @@ -388,7 +452,7 @@ This is why catch-up can safely resume from a surviving pending (§4). | Dump trait + wire format | [`format.md`](format.md); `sequencer-core/src/application/`, `examples/app-core/` | | Storage (SQLite only) | `sequencer/src/storage/snapshot_dumps.rs`; atomic close + promote in `storage/ingress.rs` | | Lane integration (take/observe/GC) | `sequencer/src/ingress/inclusion_lane/snapshot.rs`, `mod.rs`, `catch_up.rs` | -| Runtime startup sequence | `sequencer/src/runtime/workers.rs` | +| Runtime startup sequence | `sequencer/src/commands/run/startup_hygiene.rs` (called from `commands/run/workers.rs`) | | HTTP serving + leases | `sequencer/src/egress/api/snapshot.rs` | | Recovery clear | `sequencer/src/storage/recovery.rs` | diff --git a/docs/threat-model/README.md b/docs/threat-model/README.md index 68a8f7e0..44173533 100644 --- a/docs/threat-model/README.md +++ b/docs/threat-model/README.md @@ -19,11 +19,11 @@ What we are protecting: |-------|-------|--------------| | InputBox contract | Trusted | Authenticates `msg_sender` on `addInput`. Must be rollups-contracts **v3+**: the reader anchors its scan at the application's deployment block, sound only because the v3 InputBox reverts `addInput` for not-yet-deployed apps. Bootstrap witnesses this via `version()` and refuses pre-v3 boxes. Use correctly; do not model forgery. | | Our Ethereum node | Trusted, fail-stop | Inside our infra. May become unreachable; will never lie. | -| RPC endpoint (`CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT`) | Trusted, fail-stop — **must be one consistent node** | The code supports exactly **one** endpoint, shared by reader, submitter, poster, flusher, and fee oracle; no fallback tier exists yet. Behind a load-balanced fleet, lagging replicas can silently truncate `get_logs` ranges and desynchronize the flush/re-sync views (review F5/F2). The reader now fails loud on an incomplete InputAdded set (F5): a per-app index contiguity check (right prefix) plus a `getNumberOfInputs` count witness pinned at the scanned safe block (complete prefix), so a dropped/clamped/truncated-tail input is detected before the safe head advances rather than silently skipped, and recovery refuses if the re-sync lags the flush view (F2). (The count witness is fetched before the scan; when it matches the stored input count the reader advances the safe head without a `get_logs` crawl — same trust base, since a fail-stop node's pinned-block count cannot understate without lying.) The residual fleet exposure is a node that lies *consistently* about both its logs and its input count — outside the fail-stop model. A semi-trusted fallback tier (Infura/Alchemy) is future work. | -| Operator env / CLI flags | Trusted | Setup configuration is authoritative — including the reviewed Uniswap V3 WETH/fee-token pool the fee oracle quotes. The complete source is pinned in deployment identity; run cannot replace it. | -| Uniswap V3 pool (fee oracle) | Semi-trusted L1 state | Spot manipulation of a deep pool is mitigated by a 30-minute TWAP plus 10× slack in `batch_policy.log_slack`. Residual risks: TWAP lag during real moves and thin/wrong pool misconfiguration. Multi-hop pricing is out of scope. Setup writes the first Uniswap quote (same hard L1 requirement as the rest of setup). `run` tolerates transient connect/refresh failures and continues from the persisted `batch_policy.log_gas_price`, matching the warm-boot policy for unreachable RPC once identity is pinned; non-transient misconfig (`WrongTokenPair`, `MissingPoolCode`, chain-id mismatch) stays terminal. Freshness is bounded by a persisted `log_gas_price_updated_at_ms` stamped on every successful write, enforced at boot and in `run_forever` against the same max-age as L1 read-staleness (`l1_read_stale_after_blocks * seconds_per_block`). A pool/`observe` failure while the RPC and input reader stay healthy therefore does **not** trip the L1 stale-view danger detector — under the trusted-pool assumption that is accepted residual risk until the fee-oracle max-age fires. | +| RPC endpoint (`CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT`) | Trusted, fail-stop — **must be one consistent node** | The code supports exactly **one** endpoint, shared by reader, submitter, poster, flusher, and fee oracle; no fallback tier exists yet. Behind a load-balanced fleet, lagging replicas can silently truncate `get_logs` ranges and desynchronize the flush/re-sync views. The reader fails loud on an incomplete InputAdded set: a per-app index contiguity check (right prefix) plus a `getNumberOfInputs` count witness pinned at the scanned safe block (complete prefix), so a dropped/clamped/truncated-tail input is detected before the safe head advances rather than silently skipped, and recovery refuses if the re-sync lags the flush view. (The count witness is fetched before the scan; when it matches the stored input count the reader advances the safe head without a `get_logs` crawl — same trust base, since a fail-stop node's pinned-block count cannot understate without lying.) The residual fleet exposure is a node that lies *consistently* about both its logs and its input count — outside the fail-stop model. A semi-trusted fallback tier (Infura/Alchemy) is future work. | +| Operator env / CLI flags | Trusted, **mistakes foot-gun-guarded** | Setup configuration is authoritative — including the reviewed Uniswap V3 WETH/fee-token pool the fee oracle quotes. The complete source is pinned in deployment identity; run cannot replace it. The operator is trusted, not infallible: the supported operator-mistake class is *accidental concurrent or stale use of one data directory* — two processes on one dir (kernel process lock), a mistyped `--data-dir` (open refuses paths with no database). Deliberate operator subversion, copied-directory coordination, and distributed fencing remain out of scope (see the ADR's non-goals); mechanisms defending this class are judged against this boundary rather than re-litigated per review. | +| Uniswap V3 pool (fee oracle) | Semi-trusted L1 state | Spot manipulation of a deep pool is mitigated by a 30-minute TWAP plus 10× slack in `batch_policy.log_slack`. Residual risks: TWAP lag during real moves and thin/wrong pool misconfiguration. Multi-hop pricing is out of scope. Setup writes the first Uniswap quote under the same hard L1 requirement as the rest of setup; a failed quote leaves setup incomplete. `run` does not gate recovery or admission on another quote: it starts from the persisted `batch_policy.log_gas_price`, constructs the source from setup-validated identity without RPC, and launches a refresher that logs/retries transient quote failures indefinitely while retaining that price. `log_gas_price_updated_at_ms` records successful observation for telemetry; it is not an expiry gate. A shared-endpoint outage or stale view is already caught by safe-head progress, while a pool/`observe`-specific failure with a healthy input reader is accepted as an unbounded economic residual: stale-low pricing can subsidize DA/weaken the fee spam barrier and stale-high pricing can reject users, but neither changes canonical execution because the frame's persisted fee is immutable and enforced by both sides. The 10× slack is a margin, not a proof against arbitrary market movement. Deterministic setup-time source misconfiguration (`WrongTokenPair`, `MissingPoolCode`, chain-id mismatch), fatal arithmetic, and persistent storage faults remain terminal. | | Batch-submitter private key | Private | Held in operator infra. Not reachable by the network. | -| Sequencer's own code | Trusted (bug-free is a precondition) | Bugs are caught via tests and review, not defended against at runtime. See "self-trust" below. | +| Sequencer's own code | Trusted (bug-free is a precondition) | Bugs are prevented through tests/review and contained by fail-loud runtime invariant checks; they are not treated as adversarial behavior that the protocol can recover around. See "self-trust" below. | | **L1 mempool and block builders** | **Fully adversarial** | May reorder, delay, drop, or selectively include submitted transactions. Private mempools mean "dropped" is indistinguishable from "delayed indefinitely." | | HTTP clients at `POST /tx` | Untrusted | Arbitrary public callers. May submit malformed, malicious, or replay payloads. | | WebSocket subscribers at `/ws/subscribe` | Internal, but untrusted for data-exposure | Intended for internal indexers. Treat as public for what is exposed. | @@ -31,9 +31,9 @@ What we are protecting: ### Self-trust -The sequencer trusts its own code in a specific sense: **impossible states are never *handled*.** There are no graceful fallback paths, no re-validation of a neighbor module's answer, no code that keeps running past a violated internal contract. If the sequencer emits a malformed batch, frame, or user op, it is in a bug state that requires manual intervention; recovery addresses liveness failures (infrastructure outages, network partitions, gateway failure), not bug-induced malformed state. +The sequencer trusts its own code in a specific sense: **impossible states are never *handled*.** There are no graceful fallback paths, no re-validation of a neighbor module's answer, no code that keeps running past a violated internal contract. If the sequencer emits a malformed batch, frame, or user op, it is in a bug state that requires manual intervention; normal preemptive recovery addresses liveness failures (infrastructure outages, network partitions, gateway failure), not bug-induced malformed state. Cockroach recovery is the separate operator-directed rebuild path when durable state cannot be trusted. -This is **not** a prohibition on checking. Internal invariants are enforced loudly wherever a check is near-free — the type system, SQL constraints and triggers, boundary assertions — because in this system a loud crash is recoverable by design (orchestrator respawn + startup recovery), while a silently-tolerated bug that externalizes (a signed batch, an ack, a feed event) is state divergence: as severe as theft and undefendable at runtime. The rule, in short: **assert real invariants, fail loud, never absorb silently, never handle gracefully.** The decision test and the register of cross-module invariants live in [`docs/invariants.md`](../invariants.md). +This is **not** a prohibition on checking. Internal invariants are enforced loudly wherever a check is near-free — the type system, SQL constraints and triggers, boundary assertions — because failing loud preserves safety, while a silently-tolerated bug that externalizes (a signed batch, an ack, a feed event) is state divergence: as severe as theft and undefendable at runtime. Loud failure is not automatically self-healing: transient faults may clear on restart, but persistent invalid state is terminal and may require inspection or cockroach recovery. The rule, in short: **assert real invariants, fail loud, never absorb silently, never handle gracefully.** The decision test and the register of cross-module invariants live in [`docs/invariants.md`](../invariants.md). Inputs from untrusted actors are validated rigorously, as ever. @@ -41,12 +41,28 @@ Inputs from untrusted actors are validated rigorously, as ever. - L1 provider outages (primary and fallback), minutes to hours - Process crashes at arbitrary points, including mid-transaction +- **Restart after a terminal exit (accepted residual window).** There + is no boot gate on a prior terminal verdict: a deliberate restart after + exit 30 boots through the fact-derived reducer. Every fault whose evidence + the boot path reads re-refuses before the first soft confirmation — + canonical divergence (persisted fact), misconfiguration (re-checked every + boot), boot-path storage corruption, incomplete setup — and the + batch/frame spine is re-inspected by the runtime danger detector within + seconds of launch. The accepted residual is narrow: corrupt payload bytes + in rows at/below the lane's resume checkpoint re-trip only when the WS + feed pages them (bounded by its catch-up window) or the submitter + re-encodes a pending batch, and a fault with no durable evidence (a panic + whose trigger does not recur) does not re-trip at all. The window is + entered only by a deliberate operator restart after an exit-30 page, and + it is bounded by backstops that never depended on a boot gate: + rollbackable soft confirmations, the watchdog byte-compare, and the I15 + divergence freeze. - **Adversarial mempool:** reorder, delay, drop, selective inclusion by builders -- **Zombie transactions:** a submitted batch may sit in a private mempool indefinitely and land long after we believed it was gone. Two load-bearing defenses: the recovery flusher consumes every wallet-nonce slot this deployment ever used (anchored by the persisted watermark, review R1a) so zombies cannot claim them; and the content-identity check (review R2) compares every *accepted* landing against the batch we sealed at that nonce — a zombie that lands anyway is detected within one safe-finality delay and the node refuses into cockroach recovery. This is trust-boundary validation of external input (the mempool replaying our own stale transactions at times we don't control), not defense-in-depth against self-bugs. +- **Zombie transactions:** a submitted batch may sit in a private mempool indefinitely and land long after we believed it was gone. Two load-bearing defenses: the recovery flusher consumes every wallet-nonce slot this deployment ever used (anchored by the persisted watermark, I14) so zombies cannot claim them; and the content-identity check (I9/I15) compares every at/above-anchor *simulated-accepted* landing against the valid closed batch we sealed at that nonce. A foreign or byte-different landing records divergence when it becomes safe and is ingested, freezes the accepted frontier, and requires cockroach recovery. This is trust-boundary validation of external input (the mempool replaying our own stale transactions at times we don't control), not defense-in-depth against self-bugs or a general canonical-state oracle. In cockroach recovery the watermark does not survive the wipe, so that flush is best-effort by construction; the content-identity check is what keeps the residual zombie detected-and-frozen rather than silent (see `docs/recovery/cockroach.md`, step 2). - L1 reorgs up to safe depth - Malicious `POST /tx` callers: malformed signatures, spoofed sender, replay across chains or apps, nonce manipulation - Malicious direct-input senders: arbitrary payload, any intent; sender authenticity is guaranteed by InputBox -- Scheduler/sequencer protocol divergence of any kind (ordering, nonce rules, signature validity, fee semantics) +- Scheduler/sequencer protocol divergence of any kind (ordering, nonce rules, signature validity, fee semantics) is an in-scope correctness consequence. The content-identity check detects accepted-batch identity failures only; there is no complete runtime detector for the broader class. Shared semantics, review, and tests are preventative, and cockroach recovery is the remedy only after another signal or operator investigation diagnoses divergence. ## Out of scope @@ -74,14 +90,14 @@ This assumes a **known, bounded-variance relationship** between elapsed wall-clo 1. **Known average block time** — `CARTESI_SEQUENCER_SECONDS_PER_BLOCK` (default 12s, Ethereum mainnet) accurately reflects the target chain's block cadence. 2. **Bounded variance** — over the danger-threshold window (~4h on mainnet), the delta between `elapsed_seconds / avg_block_time` and actual mined blocks is small. On Ethereum mainnet this holds: slot proposers occasionally skip, but >99% of slots produce a block. -3. **Wall clock is monotonic and accurate** — the host's `SystemTime::now()` does not jump backward significantly or drift. Handled by saturating subtraction against clock backward jumps, but not against systematic drift. +3. **Wall clock is accurate enough for elapsed-time estimation.** A discrete jump of a full block-time or more against either persisted safety baseline is detected and makes the L1 view unusable until the clock or a new safe-head observation catches up; sub-block skew is tolerated as quantization noise, and the fault is evaluated only after the observed-safe danger checks. Gradual or systematic drift remains an external assumption. -**Where it matters.** Only on the fallback path — when L1 is unreachable and we cannot observe block numbers directly. When L1 is up, observed block numbers are authoritative and this assumption is not consulted. +**Where it matters.** The missed-block estimate is a fallback for a safe head that stops advancing, whether the RPC is unreachable or still answers with a stalled view. Clock usability is also checked whenever persisted safety baselines are aged: observed block numbers remain authoritative and their danger verdicts run first, but a clock a full block-time or more out of step with either baseline still makes that view unusable for estimation and for worker admission. **Violation modes.** - **Chain with unstable block time.** A chain where average block time drifts substantially (e.g., PoW networks under major hashrate swings) would make the estimate less reliable. Mitigation: `CARTESI_SEQUENCER_SECONDS_PER_BLOCK` should be tuned conservatively (overestimate block time → underestimate missed blocks → more cautious recovery triggers). - **Operator misconfigures `CARTESI_SEQUENCER_SECONDS_PER_BLOCK`.** Typo or copy-paste error pointing at the wrong chain's cadence. Operator-trust scope. -- **Significant host clock drift.** A sequencer host whose clock lags or leads the real-world by minutes per day could slowly desynchronize its danger estimates from reality. +- **Significant host clock drift.** A sequencer host whose clock lags or leads the real-world by minutes per day could slowly desynchronize its danger estimates from reality. A detectable backward crossing of a persisted baseline refuses operation; gradual drift may not. **Corollary for test design.** To deterministically exercise the wall-clock fallback, tests must maintain this coupling: when advancing the L1 block count, they should also advance (or simulate) the corresponding wall-clock interval. Our e2e harness does the reverse — it rewinds `l1_safe_head.synced_at_ms` to an older timestamp, which is semantically equivalent to advancing the wall clock. diff --git a/docs/watchdog/README.md b/docs/watchdog/README.md index c4658ada..18b811b2 100644 --- a/docs/watchdog/README.md +++ b/docs/watchdog/README.md @@ -361,9 +361,11 @@ See [`staging-drills.md`](staging-drills.md) for divergence signal and watchdog ## Related sequencer tests ```bash -cargo test -p sequencer snapshot_endpoints -- --test-threads=1 +cargo test -p sequencer --lib integration_tests::snapshot_endpoints -- --test-threads=1 cargo test -p app-core wallet_snapshot -- --test-threads=1 ``` -HTTP integration for snapshot routes lives in `sequencer/tests/snapshot_endpoints.rs`. +HTTP integration-style coverage for snapshot routes lives in +`sequencer/src/integration_tests/snapshot_endpoints.rs`; it stays inside the +crate so raw server launch remains crate-private. SSZ golden bytes for the toy wallet live in `tests/fixtures/wallet_snapshot_empty.{hex,bin}`. diff --git a/docs/watchdog/design-notes.md b/docs/watchdog/design-notes.md index 4c3f5872..29314090 100644 --- a/docs/watchdog/design-notes.md +++ b/docs/watchdog/design-notes.md @@ -36,6 +36,32 @@ Each tick: There is no advance-only mode. Advancing the CM is just an implementation step inside a compare cycle. +## Detection boundary: watchdog versus the content-identity check + +The watchdog is the broad independent detector for application-state +divergence at a finalized checkpoint. It is not the sequencer's own +accepted-batch wire-identity detector, and neither mechanism subsumes the +other. + +The content-identity check runs inside the input reader's atomic +safe-input sync. For every +at/above-anchor landing the mirrored scheduler accepts, it requires a +byte-identical valid local sealed batch at that nonce. A foreign or mismatched +landing persists `canonical_divergence` and structurally freezes the accepted +frontier and finalized-snapshot promotion. The offending landing therefore +normally never produces a newer `/finalized_state/inclusion_block` for the +watchdog to compare. Under the unchanged-head optimization above, a watchdog +tick legitimately exits idle. Distinct wire bytes can also be application-state +equivalent, which a byte comparison of resulting snapshots would not expose. + +Conversely, the content-identity check shares the sequencer's off-chain acceptance predicate and does +not independently replay application execution. The watchdog can catch +direct-input, user-op, scheduler, or application-state divergence outside its +narrow predicate once a comparable finalized checkpoint is published. +`DangerDetector`, not the watchdog, owns prompt process-wide reaction to the +durable divergence marker; the inclusion lane also refuses the poisoned projection +opportunistically if its existing frontier read wins first. + ## Watchdog State The watchdog state is canonical from the watchdog's point of view. The @@ -107,7 +133,7 @@ memory, while avoiding whole-range `logs` plus whole-range decoded `inputs`. The Cartesi binding may still queue one partition internally while feeding it to the machine. -## Open Questions Before Merge +## Open questions - Is the current crash model sufficient for a watchdog sidecar, or do operators need fsync/SQLite durability? diff --git a/docs/watchdog/getting-started.md b/docs/watchdog/getting-started.md index bcc60957..8a8b6c39 100644 --- a/docs/watchdog/getting-started.md +++ b/docs/watchdog/getting-started.md @@ -218,6 +218,6 @@ just doctor # toolchain sanity before CM-backed tests just test-watchdog # Lua unit tests (no live chain) just test-watchdog-e2e # CM advance/inspect (optional live sequencer URL) just test-watchdog-compare-harness # Full stack smoke -cargo test -p sequencer --test snapshot_endpoints +cargo test -p sequencer --lib integration_tests::snapshot_endpoints cargo test -p app-core wallet_snapshot ``` diff --git a/docs/watchdog/operator-deployment.md b/docs/watchdog/operator-deployment.md index b66d6adf..219c7822 100644 --- a/docs/watchdog/operator-deployment.md +++ b/docs/watchdog/operator-deployment.md @@ -363,6 +363,41 @@ with its store or prune, and omitting `--delete` **accumulates a per-block history in S3** while local disk stays at one snapshot. Restore feeds a chosen snapshot back through the watchdog/sequencer recovery workflow. +## Sequencer restart policy + +The sequencer's exit codes are the restart contract: 10 +restart-expect-recovery, 20 restart-transient, 30 terminal — **page an +operator, do not auto-restart**, 40 wipe + `setup --recovery`, 1 +unclassified restart-with-backoff. Operational notes: + +- **The exit code is the whole restart contract.** There is no database + gate and no acknowledgement command: + standard recovery is automatic on every boot, and a persistent terminal + fault re-detects fail-loud when the faulty state is next read. Configure + the supervisor to honor 30 (stop and page) — that configuration is what + bounds a crash loop. +- **Supervisor recipes.** systemd can act on the code directly: + `RestartPreventExitStatus=30 ABRT` (SIGABRT/134 is terminal-class too). + Kubernetes Deployments restart regardless of exit code, and there is no + boot gate — so on k8s a terminal exit will restart-loop through + re-detection windows, serving traffic in between. There, the crash-loop + bound is your alerting, not the restart policy: page immediately on + `lastState.terminated.exitCode == 30` (and on signal exits / 134). +- **A terminal containment that cannot drain within two seconds exits via + `abort()` (SIGABRT, status 134), not code 30.** Treat 134 from the + sequencer as terminal-class; the cause is in the logs and in the + `terminal_faults` black box when the write got through. +- **After an unclean death (OOM, node reboot, SIGKILL) no action is + needed**: the next start re-derives everything from facts. For + postmortems, the `terminal_faults` table records every terminal cause + (best-effort, append-only, traveling with the data directory — + `SELECT * FROM terminal_faults ORDER BY fault_id DESC`); an unclean + death that never reached containment leaves only the process logs. +- **Canonical divergence is the one manual path**: the sequencer freezes + the acceptance frontier, refuses all commands, and the remedy is a + fresh-directory `setup --recovery` (cockroach). You will typically learn + of it from the watchdog before the sequencer tells you. + ## Troubleshooting (live deployments) | Symptom | Likely cause | diff --git a/examples/app-core/src/application/wallet.rs b/examples/app-core/src/application/wallet.rs index 669b2755..8a00437a 100644 --- a/examples/app-core/src/application/wallet.rs +++ b/examples/app-core/src/application/wallet.rs @@ -14,7 +14,11 @@ use types::{Erc20Deposit, Erc20Transfer}; use super::MAX_METHOD_PAYLOAD_BYTES as WALLET_MAX_METHOD_PAYLOAD_BYTES; use super::Method; use super::{DepositNotice, TransferNotice}; -use sequencer_core::application::{AppError, AppOutput, AppOutputs, Application, InvalidReason}; +use sequencer_core::application::{ + AppError, AppOutput, AppOutputs, Application, ApplicationProgress, ApplyInputCapability, + InvalidReason, ProgressCommitCapability, +}; +use sequencer_core::history::ExecutedInputCount; use sequencer_core::l2_tx::ValidUserOp; use sequencer_core::user_op::UserOp; @@ -55,8 +59,7 @@ pub struct WalletApp { config: WalletConfig, balances: HashMap, nonces: HashMap, - executed_input_count: u64, - last_executed_safe_block: u64, + execution_progress: ApplicationProgress, } /// Rollups-contracts v3.0.0-alpha.6 ERC20Portal. The contracts deploy at @@ -87,26 +90,38 @@ impl WalletApp { config, balances: HashMap::new(), nonces: HashMap::new(), - executed_input_count: 0, - last_executed_safe_block: 0, + execution_progress: ApplicationProgress::default(), } } /// Reconstruct from decoded snapshot parts. Used by `crate::wallet_snapshot::decode`. + /// + /// The progress pair comes from untrusted dump bytes, so an incoherent + /// pair is a typed decode error like every other corrupt-snapshot case — + /// not a panic escaping `from_dump`'s `Result` (D10). pub(crate) fn from_snapshot_parts( config: WalletConfig, balances: HashMap, nonces: HashMap, executed_input_count: u64, last_executed_safe_block: u64, - ) -> Self { - Self { + ) -> Result { + let execution_progress = ApplicationProgress::try_new( + ExecutedInputCount::new(executed_input_count), + last_executed_safe_block, + ) + .ok_or_else(|| AppError::Internal { + reason: format!( + "snapshot progress is incoherent: zero executed inputs with \ + nonzero safe-block clock {last_executed_safe_block}" + ), + })?; + Ok(Self { config, balances, nonces, - executed_input_count, - last_executed_safe_block, - } + execution_progress, + }) } // Accessors for the canonical snapshot encoder (`crate::wallet_snapshot`). @@ -134,15 +149,14 @@ impl WalletApp { #[cfg(test)] pub(crate) fn set_executed_input_count(&mut self, count: u64) { - self.executed_input_count = count; - } - - pub(crate) fn executed_input_count(&self) -> u64 { - self.executed_input_count + self.execution_progress = ApplicationProgress::new( + ExecutedInputCount::new(count), + self.execution_progress.last_executed_safe_block(), + ); } pub fn last_executed_safe_block(&self) -> u64 { - self.last_executed_safe_block + self.execution_progress.last_executed_safe_block() } /// Deterministic JSON of the non-default logical state (debug only). @@ -208,7 +222,10 @@ impl WalletApp { } fn bump_nonce(&mut self, addr: Address) { - let next = self.expected_nonce(&addr).wrapping_add(1); + let next = self + .expected_nonce(&addr) + .checked_add(1) + .expect("wallet nonce overflow: no canonical successor"); self.nonces.insert(addr, next); } @@ -266,10 +283,11 @@ impl Application for WalletApp { Ok(()) } - fn execute_valid_user_op( + fn apply_valid_user_op( &mut self, + _capability: ApplyInputCapability<'_>, user_op: &ValidUserOp, - safe_block: u64, + _safe_block: u64, ) -> Result { let sender = user_op.sender; let fee_cost = sequencer_core::fee::fee_to_linear(user_op.fee); @@ -314,13 +332,12 @@ impl Application for WalletApp { _ => {} } - self.executed_input_count = self.executed_input_count.saturating_add(1); - self.last_executed_safe_block = self.last_executed_safe_block.max(safe_block); Ok(outputs) } - fn execute_direct_input( + fn apply_direct_input( &mut self, + _capability: ApplyInputCapability<'_>, input: &sequencer_core::l2_tx::DirectInput, ) -> Result { let mut outputs = Vec::new(); @@ -357,17 +374,18 @@ impl Application for WalletApp { } } - self.executed_input_count = self.executed_input_count.saturating_add(1); - self.last_executed_safe_block = self.last_executed_safe_block.max(input.block_number); Ok(outputs) } - fn executed_input_count(&self) -> u64 { - self.executed_input_count + fn execution_progress(&self) -> &ApplicationProgress { + &self.execution_progress } - fn last_executed_safe_block(&self) -> u64 { - self.last_executed_safe_block + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + &mut self.execution_progress } fn canonical_snapshot_bytes(&self) -> Result, AppError> { @@ -432,9 +450,10 @@ mod tests { use types::Erc20Transfer; use types::alloy_sol_types::SolCall; - use super::{WalletApp, WalletConfig}; + use super::{ApplicationProgress, ExecutedInputCount, WalletApp, WalletConfig}; use crate::application::{DepositNotice, Transfer, TransferNotice, Withdrawal}; use sequencer_core::application::{AppError, AppOutput, Application, InvalidReason}; + use sequencer_core::application::{execute_direct_input, execute_valid_user_op}; use sequencer_core::l2_tx::{DirectInput, ValidUserOp}; use sequencer_core::user_op::UserOp; @@ -480,9 +499,9 @@ mod tests { data: Vec::new(), }; let gas_cost = sequencer_core::fee::fee_to_linear(fee_exponent); - let outputs = app - .execute_valid_user_op(&valid, 0) - .expect("execute valid op"); + let outputs = execute_valid_user_op(&mut app, &valid, 0) + .expect("execute valid op") + .outputs; assert_eq!(app.current_user_nonce(sender), 1); assert_eq!(app.current_user_balance(sender), initial_balance - gas_cost); @@ -545,9 +564,9 @@ mod tests { data: ssz::Encode::as_ssz_bytes(&legacy), }; - let outputs = app - .execute_valid_user_op(&valid, 0) - .expect("execute valid user op"); + let outputs = execute_valid_user_op(&mut app, &valid, 0) + .expect("execute valid user op") + .outputs; assert_eq!(app.current_user_nonce(sender), before_sender_nonce + 1); // Gas cost of 1 unit (fee_to_linear(0) = 1) is deducted @@ -573,8 +592,9 @@ mod tests { let nested_sender = address!("0x7777777777777777777777777777777777777777"); let before = app.current_user_balance(nested_sender); - let outputs = app - .execute_direct_input(&DirectInput { + let outputs = execute_direct_input( + &mut app, + &DirectInput { sender: super::SEPOLIA_ERC20_PORTAL_ADDRESS, block_number: 123, payload: encode_erc20_deposit_payload( @@ -582,14 +602,16 @@ mod tests { nested_sender, U256::from(250_u64), ), - }) - .expect("execute deposit direct input"); + }, + ) + .expect("execute deposit direct input") + .outputs; assert_eq!( app.current_user_balance(nested_sender), before + U256::from(250_u64) ); - assert_eq!(app.executed_input_count(), 1); + assert_eq!(app.executed_input_count().get(), 1); assert_eq!(outputs.len(), 1); match &outputs[0] { AppOutput::Notice(payload) => { @@ -608,8 +630,9 @@ mod tests { let nested_sender = address!("0x7777777777777777777777777777777777777777"); let before = app.current_user_balance(nested_sender); - let outputs = app - .execute_direct_input(&DirectInput { + let outputs = execute_direct_input( + &mut app, + &DirectInput { sender: address!("0x3333333333333333333333333333333333333333"), block_number: 123, payload: encode_erc20_deposit_payload( @@ -617,11 +640,13 @@ mod tests { nested_sender, U256::from(250_u64), ), - }) - .expect("execute non-portal direct input"); + }, + ) + .expect("execute non-portal direct input") + .outputs; assert_eq!(app.current_user_balance(nested_sender), before); - assert_eq!(app.executed_input_count(), 1); + assert_eq!(app.executed_input_count().get(), 1); assert!(outputs.is_empty()); } @@ -632,8 +657,9 @@ mod tests { let unsupported_token = address!("0x9999999999999999999999999999999999999999"); let before = app.current_user_balance(nested_sender); - let outputs = app - .execute_direct_input(&DirectInput { + let outputs = execute_direct_input( + &mut app, + &DirectInput { sender: super::SEPOLIA_ERC20_PORTAL_ADDRESS, block_number: 123, payload: encode_erc20_deposit_payload( @@ -641,11 +667,13 @@ mod tests { nested_sender, U256::from(250_u64), ), - }) - .expect("unsupported token should be ignored"); + }, + ) + .expect("unsupported token should be ignored") + .outputs; assert_eq!(app.current_user_balance(nested_sender), before); - assert_eq!(app.executed_input_count(), 1); + assert_eq!(app.executed_input_count().get(), 1); assert!(outputs.is_empty()); } @@ -653,15 +681,18 @@ mod tests { fn malformed_trusted_portal_deposit_is_a_no_op() { let mut app = WalletApp::new(WalletConfig::default()); - let outputs = app - .execute_direct_input(&DirectInput { + let outputs = execute_direct_input( + &mut app, + &DirectInput { sender: super::SEPOLIA_ERC20_PORTAL_ADDRESS, block_number: 123, payload: vec![0xaa; 10], - }) - .expect("malformed trusted portal payload should be ignored"); + }, + ) + .expect("malformed trusted portal payload should be ignored") + .outputs; - assert_eq!(app.executed_input_count(), 1); + assert_eq!(app.executed_input_count().get(), 1); assert!(outputs.is_empty()); } @@ -684,9 +715,9 @@ mod tests { })), }; - let outputs = app - .execute_valid_user_op(&valid, 0) - .expect("execute transfer"); + let outputs = execute_valid_user_op(&mut app, &valid, 0) + .expect("execute transfer") + .outputs; assert_eq!( app.current_user_balance(sender), @@ -721,9 +752,9 @@ mod tests { })), }; - let outputs = app - .execute_valid_user_op(&valid, 0) - .expect("execute withdrawal"); + let outputs = execute_valid_user_op(&mut app, &valid, 0) + .expect("execute withdrawal") + .outputs; assert_eq!( app.current_user_balance(sender), @@ -778,7 +809,7 @@ mod tests { fee: fee_exponent, data: Vec::new(), }; - app.execute_valid_user_op(&valid, 0).expect("execute op"); + execute_valid_user_op(&mut app, &valid, 0).expect("execute op"); assert_eq!( app.current_user_balance(sender), @@ -808,7 +839,7 @@ mod tests { fee: fee_exponent, data: Vec::new(), }; - app.execute_valid_user_op(&valid, 0).expect("execute op"); + execute_valid_user_op(&mut app, &valid, 0).expect("execute op"); assert_eq!( app.current_user_balance(sender), @@ -831,8 +862,7 @@ mod tests { app.balances.insert(bob, U256::from(5678_u64)); app.nonces.insert(alice, 4); app.nonces.insert(bob, 9); - app.executed_input_count = 42; - app.last_executed_safe_block = 777; + app.execution_progress = ApplicationProgress::new(ExecutedInputCount::new(42), 777); let prefix = temp_dump_prefix(); app.create_dump(&prefix).expect("create dump"); @@ -855,11 +885,7 @@ mod tests { ); assert_eq!(restored.balances, app.balances); assert_eq!(restored.nonces, app.nonces); - assert_eq!(restored.executed_input_count, app.executed_input_count); - assert_eq!( - restored.last_executed_safe_block, - app.last_executed_safe_block - ); + assert_eq!(restored.execution_progress, app.execution_progress); } #[test] @@ -875,7 +901,7 @@ mod tests { fee: 0, data: Vec::new(), }; - app.execute_valid_user_op(&valid, 100).expect("execute op"); + execute_valid_user_op(&mut app, &valid, 100).expect("execute op"); assert_eq!(app.last_executed_safe_block(), 100); // A direct input advances the clock via its own inclusion block. @@ -884,7 +910,7 @@ mod tests { block_number: 150, payload: Vec::new(), }; - app.execute_direct_input(&direct).expect("execute direct"); + execute_direct_input(&mut app, &direct).expect("execute direct"); assert_eq!(app.last_executed_safe_block(), 150); // max(): an older block must never regress the clock. A direct's @@ -895,8 +921,7 @@ mod tests { block_number: 120, payload: Vec::new(), }; - app.execute_direct_input(&older_direct) - .expect("execute older direct"); + execute_direct_input(&mut app, &older_direct).expect("execute older direct"); assert_eq!(app.last_executed_safe_block(), 150); } @@ -923,7 +948,7 @@ mod tests { .insert(address!("0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"), 7); app.nonces .insert(address!("0x1111111111111111111111111111111111111111"), 8); - app.executed_input_count = 99; + app.set_executed_input_count(99); let prefix_a = temp_dump_prefix(); let prefix_b = temp_dump_prefix(); diff --git a/examples/app-core/src/wallet_snapshot.rs b/examples/app-core/src/wallet_snapshot.rs index 1c6fa2e0..d5fd23a0 100644 --- a/examples/app-core/src/wallet_snapshot.rs +++ b/examples/app-core/src/wallet_snapshot.rs @@ -17,7 +17,7 @@ use ssz::{Decode, Encode}; use ssz_derive::{Decode as SszDecode, Encode as SszEncode}; use crate::application::{WalletApp, WalletConfig}; -use sequencer_core::application::AppError; +use sequencer_core::application::{AppError, Application}; #[derive(Debug, Clone, PartialEq, Eq, SszEncode, SszDecode)] pub struct SnapshotBalance { @@ -68,7 +68,7 @@ pub fn encode(app: &WalletApp) -> Vec { sequencer_address: app.config().sequencer_address.into_array(), balances, nonces, - executed_input_count: app.executed_input_count(), + executed_input_count: app.executed_input_count().get(), last_executed_safe_block: app.last_executed_safe_block(), } .as_ssz_bytes() @@ -101,7 +101,7 @@ pub fn decode(bytes: &[u8]) -> Result { } } - Ok(WalletApp::from_snapshot_parts( + WalletApp::from_snapshot_parts( WalletConfig { erc20_portal_address: Address::from(decoded.erc20_portal_address), supported_erc20_token: Address::from(decoded.supported_erc20_token), @@ -111,7 +111,7 @@ pub fn decode(bytes: &[u8]) -> Result { nonces, decoded.executed_input_count, decoded.last_executed_safe_block, - )) + ) } #[cfg(test)] diff --git a/examples/canonical-app/src/scheduler/mod.rs b/examples/canonical-app/src/scheduler/mod.rs index 793566d5..bc07342f 100644 --- a/examples/canonical-app/src/scheduler/mod.rs +++ b/examples/canonical-app/src/scheduler/mod.rs @@ -41,7 +41,9 @@ pub fn run_scheduler_forever( payload, }; - let result = scheduler.process_input(input); + let result = scheduler + .process_input(input) + .unwrap_or_else(|err| panic!("canonical application execution failed: {err}")); for output in &result.outputs { emit_app_output(&mut rollup, output) .unwrap_or_else(|err| panic!("scheduler failed to emit app output: {err}")); diff --git a/examples/wallet-sequencer/src/bin/wallet-sequencer-devnet.rs b/examples/wallet-sequencer/src/bin/wallet-sequencer-devnet.rs index 2914cc39..0126ddc3 100644 --- a/examples/wallet-sequencer/src/bin/wallet-sequencer-devnet.rs +++ b/examples/wallet-sequencer/src/bin/wallet-sequencer-devnet.rs @@ -2,11 +2,17 @@ // SPDX-License-Identifier: Apache-2.0 (see LICENSE) use app_core::application::{WalletApp, WalletConfig}; +use std::io::IsTerminal; use tracing_subscriber::EnvFilter; #[tokio::main] async fn main() -> std::process::ExitCode { + // ANSI styling only when a person is watching. `tracing-subscriber` + // decides styling from `NO_COLOR` alone (no TTY check), so without this + // a daemon writing to a pipe, a file, or journald interleaves escape + // codes into its operator log. tracing_subscriber::fmt() + .with_ansi(std::io::stdout().is_terminal()) .with_env_filter( EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")), ) diff --git a/examples/wallet-sequencer/src/main.rs b/examples/wallet-sequencer/src/main.rs index c33d1d3a..40c53211 100644 --- a/examples/wallet-sequencer/src/main.rs +++ b/examples/wallet-sequencer/src/main.rs @@ -2,11 +2,17 @@ // SPDX-License-Identifier: Apache-2.0 (see LICENSE) use app_core::application::{WalletApp, WalletConfig}; +use std::io::IsTerminal; use tracing_subscriber::EnvFilter; #[tokio::main] async fn main() -> std::process::ExitCode { + // ANSI styling only when a person is watching. `tracing-subscriber` + // decides styling from `NO_COLOR` alone (no TTY check), so without this + // a daemon writing to a pipe, a file, or journald interleaves escape + // codes into its operator log. tracing_subscriber::fmt() + .with_ansi(std::io::stdout().is_terminal()) .with_env_filter( EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")), ) diff --git a/justfile b/justfile index 18173ae4..91ccdfee 100644 --- a/justfile +++ b/justfile @@ -61,10 +61,7 @@ test-watchdog-compare-harness: setup watchdog-lua-deps ensure-machine-image # Run sequencer tests sequentially so partition static config (init) is not shared across parallel tests. test-sequencer: - cargo test -p sequencer --lib -- --test-threads=1 - cargo test -p sequencer --test e2e_sequencer -- --test-threads=1 - cargo test -p sequencer --test ws_broadcaster -- --test-threads=1 - cargo test -p sequencer --test batch_submitter_integration -- --test-threads=1 + cargo test -p sequencer -- --test-threads=1 test-rollups-e2e: setup ensure-machine-image ensure-sepolia-machine-image just watchdog-lua-deps diff --git a/sequencer-core/Cargo.toml b/sequencer-core/Cargo.toml index edc2b389..d1ae1e37 100644 --- a/sequencer-core/Cargo.toml +++ b/sequencer-core/Cargo.toml @@ -20,6 +20,7 @@ ssz_derive = { workspace = true } [dev-dependencies] # Scheduler unit tests sign EIP-712 user-ops to exercise signature recovery. k256 = { workspace = true } +serde_json = { workspace = true } [build-dependencies] ruint = "1.17" diff --git a/sequencer-core/src/application/mod.rs b/sequencer-core/src/application/mod.rs index 5a33448c..7d1c00bd 100644 --- a/sequencer-core/src/application/mod.rs +++ b/sequencer-core/src/application/mod.rs @@ -1,6 +1,7 @@ // (c) Cartesi and individual authors (see AUTHORS) // SPDX-License-Identifier: Apache-2.0 (see LICENSE) +use crate::history::ExecutedInputCount; use crate::l2_tx::DirectInput; use crate::l2_tx::ValidUserOp; use crate::user_op::UserOp; @@ -20,19 +21,13 @@ pub enum AppError { #[derive(Debug, Clone, PartialEq, Eq)] pub enum ExecutionOutcome { - // NOTE: this is a transaction that may fail execution but still be included. - // We don't need to differentiate it now necessarily, but we can. - Included { outputs: AppOutputs }, + /// A canonical application input executed successfully. The receipt owns + /// its pre-execution history offset and any application outputs. + Included(ExecutedInput), Invalid(InvalidReason), } -impl ExecutionOutcome { - pub fn is_included(&self) -> bool { - matches!(self, Self::Included { .. }) - } -} - #[derive(Debug, Clone, PartialEq, Eq)] pub enum AppOutput { Notice(Vec), @@ -45,6 +40,103 @@ pub enum AppOutput { pub type AppOutputs = Vec; +/// Scheduler-owned progress embedded in the application's durable state. +/// +/// Application hooks own only application-specific mutation. The shared +/// execution functions below advance this value after a hook succeeds, so the +/// history coordinate and safe-block clock are not hand-maintained by every +/// application implementation. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct ApplicationProgress { + executed_input_count: ExecutedInputCount, + last_executed_safe_block: u64, +} + +impl ApplicationProgress { + /// Construct a coherent application-history boundary. + /// + /// A nonzero clock proves that at least one input executed, so it cannot + /// accompany the zero history boundary. + /// + /// # Panics + /// + /// Panics when `executed_input_count` is zero and + /// `last_executed_safe_block` is nonzero. Use [`Self::try_new`] on + /// deserialization paths, where the pair comes from untrusted bytes and + /// the caller owes a typed error, not a panic. + pub const fn new( + executed_input_count: ExecutedInputCount, + last_executed_safe_block: u64, + ) -> Self { + match Self::try_new(executed_input_count, last_executed_safe_block) { + Some(progress) => progress, + None => panic!("zero executed inputs require a zero safe-block clock"), + } + } + + /// Fallible sibling of [`Self::new`] for decode paths: `None` when the + /// pair is incoherent (zero executed inputs with a nonzero clock). + pub const fn try_new( + executed_input_count: ExecutedInputCount, + last_executed_safe_block: u64, + ) -> Option { + if executed_input_count.get() == 0 && last_executed_safe_block != 0 { + return None; + } + Some(Self { + executed_input_count, + last_executed_safe_block, + }) + } + + pub const fn executed_input_count(self) -> ExecutedInputCount { + self.executed_input_count + } + + pub const fn last_executed_safe_block(self) -> u64 { + self.last_executed_safe_block + } + + fn checked_after_input(self, safe_block: u64) -> Option { + Some(Self { + executed_input_count: self.executed_input_count.checked_next()?, + last_executed_safe_block: if self.last_executed_safe_block > safe_block { + self.last_executed_safe_block + } else { + safe_block + }, + }) + } +} + +struct CapabilitySeal; + +/// Opaque, call-scoped authority to invoke an application's raw mutation hook. +/// +/// Only the shared execution functions in this module can construct this +/// capability. Its borrowed private seal prevents application implementations +/// from safely forging or retaining it beyond the hook call. +pub struct ApplyInputCapability<'a> { + _seal: &'a CapabilitySeal, +} + +/// Opaque, call-scoped authority to commit scheduler-owned application progress. +/// +/// This is deliberately distinct from [`ApplyInputCapability`]: application +/// hooks receive authority to mutate application state, never authority to +/// overwrite the canonical history count or safe-block clock. +pub struct ProgressCommitCapability<'a> { + _seal: &'a CapabilitySeal, +} + +/// One successfully executed canonical application input. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ExecutedInput { + /// The boundary before this input executed; this is its history offset. + pub offset: ExecutedInputCount, + pub outputs: AppOutputs, +} + #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum InvalidReason { InvalidNonce { @@ -102,37 +194,61 @@ pub trait Application: Send + Sized { current_fee: u16, ) -> Result<(), InvalidReason>; - /// Execute a validated user op. `safe_block` is the covering frame's - /// safe block; the impl must advance its safe-block clock with it: - /// `clock = max(clock, safe_block)` (see - /// [`Application::last_executed_safe_block`]). - fn execute_valid_user_op( + /// Apply a validated user op's application-specific mutation. + /// + /// Callers use [`execute_valid_user_op`], never this hook directly. The + /// shared function advances [`ApplicationProgress`] only after this hook + /// returns `Ok`. The opaque capability makes that boundary structural for + /// safe Rust callers. + fn apply_valid_user_op( &mut self, + capability: ApplyInputCapability<'_>, user_op: &ValidUserOp, safe_block: u64, ) -> Result; /// Required (no default): deposits are direct-input-only, so a silent /// no-op impl would strand every deposit on L1 with no L2 credit. - /// The impl must advance its safe-block clock with - /// `input.block_number` (the direct's L1 inclusion block): - /// `clock = max(clock, block_number)`. - fn execute_direct_input(&mut self, input: &DirectInput) -> Result; + /// Callers use [`execute_direct_input`], never this hook directly. The + /// shared function advances [`ApplicationProgress`] only after this hook + /// returns `Ok`. The opaque capability makes that boundary structural for + /// safe Rust callers. + fn apply_direct_input( + &mut self, + capability: ApplyInputCapability<'_>, + input: &DirectInput, + ) -> Result; + + /// Scheduler-owned progress embedded in, and persisted with, application + /// state. Application hooks must not mutate it. + fn execution_progress(&self) -> &ApplicationProgress; + + /// Mutable access exists only for the shared execution boundary. Its + /// distinct opaque capability is never passed to application hooks, and + /// the progress type itself exposes no mutating operation. + fn execution_progress_mut( + &mut self, + capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress; /// The app's safe-block clock: the maximum block carried by any input /// this instance has executed (frame safe blocks for user ops, L1 /// inclusion blocks for direct inputs), or 0 if nothing executed. - /// Carried in execution — not a setter — so an app cannot execute and - /// forget to advance it. Recovery reads this as `A`, the safe block a - /// checkpoint state reflects; it must therefore survive - /// `create_dump`/`from_dump` round-trips. - fn last_executed_safe_block(&self) -> u64; - - /// Count of executed inputs (user ops + direct inputs). Diagnostic - /// seam: replay/catch-up tests compare live vs replayed apps with it. - /// Required (no default) for the same reason as - /// [`Application::execute_direct_input`]. - fn executed_input_count(&self) -> u64; + /// Carried in [`ApplicationProgress`] so it advances at the same shared + /// boundary as the history count. Recovery reads this as `A`, the safe + /// block a checkpoint state reflects; it must survive dump round-trips. + fn last_executed_safe_block(&self) -> u64 { + self.execution_progress().last_executed_safe_block() + } + + /// Canonical application-history boundary. Starts at zero and advances by + /// exactly one after each successful user-op or direct-input execution; an + /// application at `X` is ready to consume history input `X`. It must + /// survive dump round-trips. The planned Track 3 feed uses this value as + /// its subscription offset; the current rowid feed has not cut over yet. + fn executed_input_count(&self) -> ExecutedInputCount { + self.execution_progress().executed_input_count() + } // -------- snapshot / dump lifecycle -------- // @@ -217,6 +333,8 @@ pub fn validate_and_execute_user_op( current_fee: u16, safe_block: u64, ) -> Result { + let progress_before_validation = *app.execution_progress(); + // Protocol invariant: max_fee must cover the current frame fee. if user_op.max_fee < current_fee { return Ok(ExecutionOutcome::Invalid(InvalidReason::InvalidMaxFee { @@ -225,7 +343,13 @@ pub fn validate_and_execute_user_op( })); } - if let Err(reason) = app.validate_user_op(sender, user_op, current_fee) { + let validation = app.validate_user_op(sender, user_op, current_fee); + assert_eq!( + *app.execution_progress(), + progress_before_validation, + "validate_user_op mutated scheduler-owned application progress" + ); + if let Err(reason) = validation { return Ok(ExecutionOutcome::Invalid(reason)); } @@ -234,6 +358,326 @@ pub fn validate_and_execute_user_op( fee: current_fee, data: user_op.data.to_vec(), }; - let outputs = app.execute_valid_user_op(&valid, safe_block)?; - Ok(ExecutionOutcome::Included { outputs }) + execute_valid_user_op(app, &valid, safe_block).map(ExecutionOutcome::Included) +} + +/// Execute one already-validated user op and advance scheduler-owned progress. +pub fn execute_valid_user_op( + app: &mut A, + user_op: &ValidUserOp, + safe_block: u64, +) -> Result { + let seal = CapabilitySeal; + execute_and_advance(app, safe_block, |app| { + app.apply_valid_user_op(ApplyInputCapability { _seal: &seal }, user_op, safe_block) + }) +} + +/// Execute one direct input and advance scheduler-owned progress. +pub fn execute_direct_input( + app: &mut A, + input: &DirectInput, +) -> Result { + let seal = CapabilitySeal; + execute_and_advance(app, input.block_number, |app| { + app.apply_direct_input(ApplyInputCapability { _seal: &seal }, input) + }) +} + +fn execute_and_advance( + app: &mut A, + safe_block: u64, + apply: F, +) -> Result +where + A: Application, + F: FnOnce(&mut A) -> Result, +{ + let progress_before = *app.execution_progress(); + let progress_after = progress_before + .checked_after_input(safe_block) + .expect("executed input count overflow: no canonical successor"); + + let apply_result = apply(app); + assert_eq!( + *app.execution_progress(), + progress_before, + "application hook mutated scheduler-owned application progress" + ); + let outputs = apply_result?; + + let seal = CapabilitySeal; + *app.execution_progress_mut(ProgressCommitCapability { _seal: &seal }) = progress_after; + assert_eq!( + *app.execution_progress(), + progress_after, + "application progress commit is incoherent with its immutable accessor" + ); + + Ok(ExecutedInput { + offset: progress_before.executed_input_count(), + outputs, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + struct ProgressApp { + progress: ApplicationProgress, + commit_target: ApplicationProgress, + applied: u64, + reject: bool, + fail: bool, + mutate_progress_in_hook: bool, + misdirect_progress_commit: bool, + } + + impl ProgressApp { + fn new(count: u64) -> Self { + Self { + progress: ApplicationProgress::new(ExecutedInputCount::new(count), 0), + commit_target: ApplicationProgress::default(), + applied: 0, + reject: false, + fail: false, + mutate_progress_in_hook: false, + misdirect_progress_commit: false, + } + } + } + + impl Application for ProgressApp { + const MAX_METHOD_PAYLOAD_BYTES: usize = 0; + + fn validate_user_op( + &self, + _sender: Address, + _user_op: &UserOp, + _current_fee: u16, + ) -> Result<(), InvalidReason> { + if self.reject { + Err(InvalidReason::InvalidNonce { + expected: 1, + got: 0, + }) + } else { + Ok(()) + } + } + + fn apply_valid_user_op( + &mut self, + _capability: ApplyInputCapability<'_>, + _user_op: &ValidUserOp, + _safe_block: u64, + ) -> Result { + self.applied += 1; + if self.mutate_progress_in_hook { + self.progress = ApplicationProgress::new( + self.progress + .executed_input_count() + .checked_next() + .expect("test count"), + 99, + ); + } + if self.fail { + Err(AppError::Internal { + reason: "injected failure".to_string(), + }) + } else { + Ok(Vec::new()) + } + } + + fn apply_direct_input( + &mut self, + _capability: ApplyInputCapability<'_>, + _input: &DirectInput, + ) -> Result { + self.applied += 1; + if self.mutate_progress_in_hook { + self.progress = ApplicationProgress::new( + self.progress + .executed_input_count() + .checked_next() + .expect("test count"), + 99, + ); + } + if self.fail { + Err(AppError::Internal { + reason: "injected failure".to_string(), + }) + } else { + Ok(Vec::new()) + } + } + + fn execution_progress(&self) -> &ApplicationProgress { + &self.progress + } + + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + if self.misdirect_progress_commit { + &mut self.commit_target + } else { + &mut self.progress + } + } + + fn from_dump(_prefix: &Path) -> Result { + unreachable!("not used") + } + + fn create_dump(&self, _prefix: &Path) -> Result<(), AppError> { + unreachable!("not used") + } + + fn delete_dump(_prefix: &Path) -> Result<(), AppError> { + unreachable!("not used") + } + + fn state_file_in_dump(prefix: &Path) -> PathBuf { + prefix.join("state") + } + } + + fn user_op() -> UserOp { + UserOp { + nonce: 0, + max_fee: 0, + data: Vec::new().into(), + } + } + + #[test] + fn shared_boundaries_own_count_and_clock_progress() { + let mut app = ProgressApp::new(0); + let user = validate_and_execute_user_op(&mut app, Address::ZERO, &user_op(), 0, 9) + .expect("execute user op"); + let ExecutionOutcome::Included(user) = user else { + panic!("user op should be included") + }; + assert_eq!(user.offset, ExecutedInputCount::ZERO); + assert_eq!(app.executed_input_count(), ExecutedInputCount::new(1)); + assert_eq!(app.last_executed_safe_block(), 9); + + let direct = execute_direct_input( + &mut app, + &DirectInput { + sender: Address::ZERO, + block_number: 12, + payload: Vec::new(), + }, + ) + .expect("execute direct"); + assert_eq!(direct.offset, ExecutedInputCount::new(1)); + assert_eq!(app.executed_input_count(), ExecutedInputCount::new(2)); + assert_eq!(app.last_executed_safe_block(), 12); + } + + #[test] + fn rejection_and_error_do_not_commit_progress() { + let mut rejected = ProgressApp::new(7); + rejected.reject = true; + assert!(matches!( + validate_and_execute_user_op(&mut rejected, Address::ZERO, &user_op(), 0, 9) + .expect("validation rejection"), + ExecutionOutcome::Invalid(_) + )); + assert_eq!(rejected.executed_input_count(), ExecutedInputCount::new(7)); + assert_eq!(rejected.applied, 0); + + let mut failed = ProgressApp::new(7); + failed.fail = true; + assert!( + execute_direct_input( + &mut failed, + &DirectInput { + sender: Address::ZERO, + block_number: 12, + payload: Vec::new(), + }, + ) + .is_err() + ); + assert_eq!(failed.executed_input_count(), ExecutedInputCount::new(7)); + assert_eq!(failed.last_executed_safe_block(), 0); + } + + #[test] + fn count_exhaustion_fails_before_application_mutation() { + let mut app = ProgressApp::new(u64::MAX); + let panic = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + let _ = execute_direct_input( + &mut app, + &DirectInput { + sender: Address::ZERO, + block_number: 1, + payload: Vec::new(), + }, + ); + })); + assert!(panic.is_err()); + assert_eq!(app.applied, 0, "overflow must preflight the app hook"); + assert_eq!( + app.executed_input_count(), + ExecutedInputCount::new(u64::MAX) + ); + } + + #[test] + #[should_panic(expected = "zero executed inputs require a zero safe-block clock")] + fn zero_count_rejects_nonzero_safe_block_clock() { + let _ = ApplicationProgress::new(ExecutedInputCount::ZERO, 1); + } + + #[test] + fn hook_progress_mutation_panics_on_success_and_error() { + for fail in [false, true] { + let mut app = ProgressApp::new(1); + app.fail = fail; + app.mutate_progress_in_hook = true; + let panic = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + let _ = execute_direct_input( + &mut app, + &DirectInput { + sender: Address::ZERO, + block_number: 12, + payload: Vec::new(), + }, + ); + })); + assert!( + panic.is_err(), + "progress mutation must fail loud when hook fail={fail}" + ); + } + } + + #[test] + fn incoherent_mutable_progress_accessor_fails_after_commit() { + let mut app = ProgressApp::new(1); + app.misdirect_progress_commit = true; + let panic = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + let _ = execute_direct_input( + &mut app, + &DirectInput { + sender: Address::ZERO, + block_number: 12, + payload: Vec::new(), + }, + ); + })); + assert!( + panic.is_err(), + "incoherent progress accessors must fail loud" + ); + } } diff --git a/sequencer-core/src/history.rs b/sequencer-core/src/history.rs new file mode 100644 index 00000000..6b38e646 --- /dev/null +++ b/sequencer-core/src/history.rs @@ -0,0 +1,184 @@ +// (c) Cartesi and individual authors (see AUTHORS) +// SPDX-License-Identifier: Apache-2.0 (see LICENSE) + +//! External history identity and version coordinates. + +use serde::{Deserialize, Serialize}; +use std::fmt; +use thiserror::Error; + +/// Boundary before the next canonical application input executes. +/// +/// If the application is at `X`, history entry `X` is the next input and a +/// successful execution moves the boundary to `X + 1`. Arithmetic is kept +/// behind checked methods so this coordinate cannot be confused with a +/// physical SQLite cursor or advanced with wrapping/saturating math. +#[derive( + Debug, Clone, Copy, Default, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize, +)] +#[serde(transparent)] +pub struct ExecutedInputCount(u64); + +impl ExecutedInputCount { + pub const ZERO: Self = Self(0); + + pub const fn new(value: u64) -> Self { + Self(value) + } + + pub const fn get(self) -> u64 { + self.0 + } + + pub const fn checked_next(self) -> Option { + self.checked_add(1) + } + + pub const fn checked_add(self, delta: u64) -> Option { + match self.0.checked_add(delta) { + Some(value) => Some(Self(value)), + None => None, + } + } +} + +/// One durable setup/rebuild era. +/// +/// The bytes must carry the RFC 4122 UUIDv4 version and variant bits. Display +/// uses the canonical lowercase hyphenated representation. The wire (text / +/// JSON) codec deliberately does not exist yet: Track 3 owns the wire +/// projection and adds it beside its consumer when that lands. +#[derive(Clone, Copy, PartialEq, Eq, Hash)] +pub struct EraId([u8; 16]); + +impl EraId { + pub const BYTE_LEN: usize = 16; + + pub fn from_bytes(bytes: [u8; Self::BYTE_LEN]) -> Result { + if bytes[6] >> 4 != 4 { + return Err(EraIdParseError::NotVersion4); + } + if bytes[8] >> 6 != 2 { + return Err(EraIdParseError::InvalidVariant); + } + Ok(Self(bytes)) + } + + pub const fn as_bytes(&self) -> &[u8; Self::BYTE_LEN] { + &self.0 + } +} + +impl TryFrom<&[u8]> for EraId { + type Error = EraIdParseError; + + fn try_from(value: &[u8]) -> Result { + let bytes: [u8; Self::BYTE_LEN] = + value + .try_into() + .map_err(|_| EraIdParseError::InvalidByteLength { + actual: value.len(), + })?; + Self::from_bytes(bytes) + } +} + +impl fmt::Display for EraId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + for (index, byte) in self.0.iter().enumerate() { + if matches!(index, 4 | 6 | 8 | 10) { + f.write_str("-")?; + } + write!(f, "{byte:02x}")?; + } + Ok(()) + } +} + +impl fmt::Debug for EraId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_tuple("EraId").field(&self.to_string()).finish() + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Error)] +pub enum EraIdParseError { + #[error("era id blob has length {actual}, expected 16")] + InvalidByteLength { actual: usize }, + #[error("era id is not UUID version 4")] + NotVersion4, + #[error("era id has a non-RFC-4122 UUID variant")] + InvalidVariant, +} + +/// Monotonic soft-history revision within one [`EraId`]. +#[derive( + Debug, Clone, Copy, Default, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize, +)] +#[serde(transparent)] +pub struct RecoveryGeneration(u64); + +impl RecoveryGeneration { + pub const fn new(value: u64) -> Self { + Self(value) + } + + pub const fn get(self) -> u64 { + self.0 + } +} + +/// Equality/discontinuity token for locally available application history. +/// Like [`EraId`], its wire form is Track 3's to define beside its consumer. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub struct HistoryVersion { + pub era_id: EraId, + pub recovery_generation: RecoveryGeneration, +} + +#[cfg(test)] +mod tests { + use super::*; + + const CANONICAL: &str = "550e8400-e29b-41d4-a716-446655440000"; + const CANONICAL_BYTES: [u8; 16] = [ + 0x55, 0x0e, 0x84, 0x00, 0xe2, 0x9b, 0x41, 0xd4, 0xa7, 0x16, 0x44, 0x66, 0x55, 0x44, 0x00, + 0x00, + ]; + + #[test] + fn era_id_displays_canonical_lowercase_hyphenated_form() { + let era = EraId::from_bytes(CANONICAL_BYTES).expect("canonical UUIDv4"); + assert_eq!(era.to_string(), CANONICAL); + } + + #[test] + fn era_id_rejects_non_v4_bytes() { + let mut not_v4 = CANONICAL_BYTES; + not_v4[6] = 0x31; + assert_eq!(EraId::from_bytes(not_v4), Err(EraIdParseError::NotVersion4)); + let mut bad_variant = CANONICAL_BYTES; + bad_variant[8] = 0x07; + assert_eq!( + EraId::from_bytes(bad_variant), + Err(EraIdParseError::InvalidVariant) + ); + assert_eq!( + EraId::try_from(&[0_u8; 15][..]), + Err(EraIdParseError::InvalidByteLength { actual: 15 }) + ); + } + + #[test] + fn executed_input_count_advances_checked() { + assert_eq!( + ExecutedInputCount::ZERO.checked_next(), + Some(ExecutedInputCount::new(1)) + ); + assert_eq!(ExecutedInputCount::new(u64::MAX).checked_next(), None); + assert_eq!( + ExecutedInputCount::new(7).checked_add(5), + Some(ExecutedInputCount::new(12)) + ); + } +} diff --git a/sequencer-core/src/lib.rs b/sequencer-core/src/lib.rs index fe615b52..7c521333 100644 --- a/sequencer-core/src/lib.rs +++ b/sequencer-core/src/lib.rs @@ -9,6 +9,7 @@ pub mod application; pub mod batch; pub mod broadcast; pub mod fee; +pub mod history; pub mod l2_tx; pub mod protocol; pub mod scheduler; diff --git a/sequencer-core/src/protocol.rs b/sequencer-core/src/protocol.rs index e12e7260..549c4ef6 100644 --- a/sequencer-core/src/protocol.rs +++ b/sequencer-core/src/protocol.rs @@ -66,6 +66,34 @@ pub enum ProtocolTimingError { read_stale_after: u64, danger_threshold: u64, }, + /// A zero block-time estimate makes the wall-clock fallback undefined. + #[error("seconds_per_block must be greater than zero")] + SecondsPerBlockZero, + /// The configured read-staleness horizon must fit in the `u64` Unix-time + /// arithmetic used by the detector. + #[error( + "l1_read_stale_after_blocks ({read_stale_after}) * seconds_per_block \ + ({seconds_per_block}) exceeds u64::MAX seconds" + )] + ReadStaleWindowOverflow { + read_stale_after: u64, + seconds_per_block: u64, + }, +} + +/// The local wall clock predates the persisted safe-head progress baseline. +/// +/// This is an unusable timing environment, not a storage invariant violation: +/// callers must stop issuing soft confirmations until the clock catches up or +/// a new safe-head advance establishes a usable baseline. +#[derive(Debug, Error, PartialEq, Eq)] +#[error( + "current wall clock ({now_ms} ms) predates last safe-head progress \ + ({last_safe_progress_ms} ms)" +)] +pub struct WallClockRegression { + pub now_ms: u64, + pub last_safe_progress_ms: u64, } /// Time-based protocol parameters: scheduler-mirroring `max_wait_blocks` @@ -95,6 +123,17 @@ pub struct ProtocolTiming { } impl ProtocolTiming { + /// Advance logical frame time after this many newly-safe L1 blocks. + /// + /// Protocol-visible, deliberately not configurable: user ops validate at + /// their frame's safe block, so this value is the application-clock + /// granularity (about a minute on mainnet), a product semantics decision + /// — not a lane implementation detail (prose owner: + /// `docs/protocol/scheduler-semantics.md`, frame-clock policy). The + /// observed tip is used directly, so delayed or epoch-sized observations + /// create one frame and never synthesize missed intermediate ticks. + pub const FRAME_CLOCK_INTERVAL_SAFE_BLOCKS: u64 = 5; + /// Validated constructor. Rejects timing configurations that would /// produce an unusable danger threshold or a degenerate margin. /// @@ -119,6 +158,9 @@ impl ProtocolTiming { if l1_read_stale_after_blocks == 0 { return Err(ProtocolTimingError::ReadStaleAfterZero); } + if seconds_per_block == 0 { + return Err(ProtocolTimingError::SecondsPerBlockZero); + } let danger_threshold = max_wait_blocks - preemptive_margin_blocks; if l1_read_stale_after_blocks >= danger_threshold { return Err(ProtocolTimingError::ReadStaleAfterPastDanger { @@ -126,6 +168,12 @@ impl ProtocolTiming { danger_threshold, }); } + l1_read_stale_after_blocks + .checked_mul(seconds_per_block) + .ok_or(ProtocolTimingError::ReadStaleWindowOverflow { + read_stale_after: l1_read_stale_after_blocks, + seconds_per_block, + })?; Ok(Self { max_wait_blocks, preemptive_margin_blocks, @@ -136,31 +184,89 @@ impl ProtocolTiming { /// The block-age threshold at which preemptive recovery triggers. /// - /// `saturating_sub` keeps this infallible even on a directly-constructed - /// `ProtocolTiming` with an invalid margin (returns 0 in that case). /// Production code goes through [`ProtocolTiming::try_new`], which rejects - /// that configuration up front. + /// an invalid margin up front. A directly-constructed invalid value is a + /// test/programming bug and fails loud here. pub fn danger_threshold(&self) -> u64 { - self.max_wait_blocks - .saturating_sub(self.preemptive_margin_blocks) + assert!( + self.preemptive_margin_blocks > 0, + "ProtocolTiming must be constructed with a nonzero preemptive margin" + ); + assert!( + self.preemptive_margin_blocks < self.max_wait_blocks, + "ProtocolTiming must be constructed with margin < max_wait_blocks" + ); + self.max_wait_blocks - self.preemptive_margin_blocks } /// Wall-clock age, in seconds, after which the L1 safe block is too old /// for the sequencer to trust its L1 view. pub fn l1_read_stale_after_secs(&self) -> u64 { + assert!( + self.l1_read_stale_after_blocks > 0, + "ProtocolTiming must be constructed with a nonzero L1 read-staleness window" + ); + assert!( + self.seconds_per_block > 0, + "ProtocolTiming must be constructed with nonzero seconds_per_block" + ); self.l1_read_stale_after_blocks - .saturating_mul(self.seconds_per_block.max(1)) + .checked_mul(self.seconds_per_block) + .expect("validated L1 read-staleness window must fit in u64 seconds") } /// Whether the safe block timestamp is too old to support recovery or /// continued soft confirmations. `None` means the view is unknown and is /// treated as unusable. pub fn l1_view_is_stale(&self, safe_block_timestamp_secs: Option, now_ms: u64) -> bool { + assert!( + self.seconds_per_block > 0, + "ProtocolTiming must be constructed with nonzero seconds_per_block" + ); let Some(timestamp_secs) = safe_block_timestamp_secs else { return true; }; let now_secs = now_ms / 1000; - now_secs.saturating_sub(timestamp_secs) >= self.l1_read_stale_after_secs() + let Some(age_secs) = now_secs.checked_sub(timestamp_secs) else { + // Ahead-of-clock is not staleness — the observation is the + // freshest possible. Whether the local *clock* is usable against + // it is a separate fault ([`Self::clock_cannot_age_l1_view`]), + // deliberately checked after the observed danger arms. + return false; + }; + age_secs >= self.l1_read_stale_after_secs() + } + + /// Whether the local clock is a full block-time or more behind the + /// persisted L1 safe-block timestamp — i.e. it cannot age the view at + /// all. Sub-block ahead-ness is ordinary NTP-scale skew against a + /// block-granular timestamp (usable, age 0) and is tolerated. + /// + /// This is a clock fault, not a view fault: `check_danger` evaluates it + /// only after the observed danger arms, which are pure block arithmetic + /// and must not be suppressed by a broken local clock. + pub fn clock_cannot_age_l1_view( + &self, + safe_block_timestamp_secs: Option, + now_ms: u64, + ) -> bool { + assert!( + self.seconds_per_block > 0, + "ProtocolTiming must be constructed with nonzero seconds_per_block" + ); + let Some(timestamp_secs) = safe_block_timestamp_secs else { + return false; + }; + // Compare at millisecond precision: flooring `now_ms` to seconds + // would make the full-block refusal fire up to 999 ms early, + // contradicting the sub-block tolerance. Saturating multiplies are + // safe: a saturated ahead-side only widens toward refusal for + // timestamps that are absurd anyway. + let timestamp_ms = timestamp_secs.saturating_mul(1000); + match timestamp_ms.checked_sub(now_ms) { + Some(ahead_ms) => ahead_ms >= self.seconds_per_block.saturating_mul(1000), + None => false, + } } /// Wall-clock-adjusted danger threshold, used when the L1 safe head may be @@ -175,21 +281,46 @@ impl ProtocolTiming { /// - Less than one block-time has elapsed — adjustment would be 0, so the /// strict check covers this case directly. /// - /// Returns `Some(adjusted)` where + /// Returns `Ok(Some(adjusted))` where /// `adjusted = danger_threshold − (elapsed_secs / seconds_per_block)`, - /// saturating at 0. + /// saturating at 0. A `now_ms` less than one block-time behind the + /// baseline is treated as elapsed 0 — the estimate quantizes elapsed time + /// into whole blocks, so a sub-block regression cannot change it and is + /// ordinary clock-step noise. Returns [`WallClockRegression`] only + /// for larger regressions, which genuinely invalidate the extrapolation; + /// callers must treat that environment as unusable rather than silently + /// reporting no elapsed time. pub fn wall_clock_adjusted_danger_threshold( &self, last_safe_progress_ms: Option, now_ms: u64, - ) -> Option { - let last = last_safe_progress_ms?; - let elapsed_secs = now_ms.saturating_sub(last) / 1000; - let missed = elapsed_secs / self.seconds_per_block.max(1); + ) -> Result, WallClockRegression> { + assert!( + self.seconds_per_block > 0, + "ProtocolTiming must be constructed with nonzero seconds_per_block" + ); + let Some(last) = last_safe_progress_ms else { + return Ok(None); + }; + let elapsed_ms = match now_ms.checked_sub(last) { + Some(elapsed) => elapsed, + None => { + let regression_secs = (last - now_ms) / 1000; + if regression_secs < self.seconds_per_block { + return Ok(None); + } + return Err(WallClockRegression { + now_ms, + last_safe_progress_ms: last, + }); + } + }; + let elapsed_secs = elapsed_ms / 1000; + let missed = elapsed_secs / self.seconds_per_block; if missed == 0 { - return None; + return Ok(None); } - Some(self.danger_threshold().saturating_sub(missed)) + Ok(Some(self.danger_threshold().saturating_sub(missed))) } /// Scheduler's staleness predicate: a batch is stale when @@ -295,7 +426,9 @@ pub fn advance_expected_batch_nonce( ) -> u64 { for nonce in observed_nonces { if nonce == expected { - expected = expected.saturating_add(1); + expected = expected + .checked_add(1) + .expect("expected batch nonce overflow: contract-impossible"); } } expected @@ -378,11 +511,61 @@ mod tests { assert!(cfg.l1_view_is_stale(Some(timestamp_secs), at)); } + #[test] + fn l1_view_is_stale_never_fires_on_a_future_timestamp() { + // Ahead-of-clock is not staleness — the observation is the freshest + // possible. Clock usability is `clock_cannot_age_l1_view`'s job. + assert!(!timing().l1_view_is_stale(Some(1_001), 1_000_000)); + assert!(!timing().l1_view_is_stale(Some(1_012), 1_000_000)); + assert!( + !timing().l1_view_is_stale(Some(1_001), 1_001_000), + "equality is a usable zero-age observation" + ); + } + + #[test] + fn clock_cannot_age_l1_view_tolerates_sub_block_skew() { + let cfg = timing(); + // No view at all: nothing for the clock to age — not a clock fault. + assert!(!cfg.clock_cannot_age_l1_view(None, 1_000_000)); + // Behind or equal: ordinary aging, never a clock fault. + assert!(!cfg.clock_cannot_age_l1_view(Some(900), 1_000_000)); + assert!(!cfg.clock_cannot_age_l1_view(Some(1_000), 1_000_000)); + // Sub-block ahead-ness is NTP-scale skew against a block-granular + // timestamp: usable, age 0. + assert!(!cfg.clock_cannot_age_l1_view(Some(1_011), 1_000_000)); + // Millisecond precision at the boundary: 11.999 s and 11.001 s ahead + // are both sub-block; exactly 12.000 s refuses. + assert!(!cfg.clock_cannot_age_l1_view(Some(1_012), 1_000_001)); + assert!(!cfg.clock_cannot_age_l1_view(Some(1_012), 1_000_999)); + assert!(cfg.clock_cannot_age_l1_view(Some(1_012), 1_000_000)); + } + + #[test] + #[should_panic(expected = "nonzero seconds_per_block")] + fn l1_view_is_stale_fails_loud_on_zero_block_time_direct_construction() { + let cfg = ProtocolTiming { + seconds_per_block: 0, + ..timing() + }; + let _ = cfg.l1_view_is_stale(Some(1_001), 1_000_000); + } + + #[test] + #[should_panic(expected = "nonzero seconds_per_block")] + fn clock_cannot_age_l1_view_fails_loud_on_zero_block_time_direct_construction() { + let cfg = ProtocolTiming { + seconds_per_block: 0, + ..timing() + }; + let _ = cfg.clock_cannot_age_l1_view(Some(1_001), 1_000_000); + } + #[test] fn wall_clock_adjusted_threshold_returns_none_without_baseline() { assert_eq!( timing().wall_clock_adjusted_danger_threshold(None, 1_000_000), - None, + Ok(None), ); } @@ -393,7 +576,39 @@ mod tests { let now = last + 11_000; assert_eq!( timing().wall_clock_adjusted_danger_threshold(Some(last), now), - None, + Ok(None), + ); + } + + #[test] + fn wall_clock_adjusted_threshold_rejects_regression() { + let last = 1_000_000; + // Sub-block regressions are quantization noise: identical outcome to + // elapsed = 0 for a block-granular estimate (clock steps are + // legitimate). + assert_eq!( + timing().wall_clock_adjusted_danger_threshold(Some(last), last - 1), + Ok(None), + "a sub-block clock step must not be a fault" + ); + assert_eq!( + timing().wall_clock_adjusted_danger_threshold(Some(last), last - 11_999), + Ok(None), + "just under one block-time of regression is still noise" + ); + // A regression of one block-time or more genuinely invalidates the + // extrapolation. + assert_eq!( + timing().wall_clock_adjusted_danger_threshold(Some(last), last - 12_000), + Err(WallClockRegression { + now_ms: last - 12_000, + last_safe_progress_ms: last, + }), + ); + assert_eq!( + timing().wall_clock_adjusted_danger_threshold(Some(last), last), + Ok(None), + "equality is a usable zero-elapsed baseline" ); } @@ -405,7 +620,7 @@ mod tests { let cfg = timing(); assert_eq!( cfg.wall_clock_adjusted_danger_threshold(Some(last), now), - Some(cfg.danger_threshold() - 25), + Ok(Some(cfg.danger_threshold() - 25)), ); } @@ -416,27 +631,40 @@ mod tests { let now = u64::MAX / 2; assert_eq!( timing().wall_clock_adjusted_danger_threshold(Some(last), now), - Some(0), + Ok(Some(0)), ); } #[test] - fn danger_threshold_saturates_to_zero_on_invalid_margin() { - // try_new rejects this configuration; if a test ever constructs it - // directly via struct-literal syntax, danger_threshold returns 0 - // rather than panicking. (Cleaner than a hard panic during a logging - // macro on production startup.) + #[should_panic(expected = "ProtocolTiming must be constructed with margin < max_wait_blocks")] + fn danger_threshold_fails_loud_on_invalid_direct_construction() { + // Production uses try_new. A direct invalid struct literal is a + // programming error and must not be normalized into threshold zero. let cfg = ProtocolTiming { preemptive_margin_blocks: MAX_WAIT, ..timing() }; - assert_eq!(cfg.danger_threshold(), 0); + let _ = cfg.danger_threshold(); + } + #[test] + #[should_panic(expected = "nonzero preemptive margin")] + fn danger_threshold_fails_loud_on_zero_margin_direct_construction() { let cfg = ProtocolTiming { - preemptive_margin_blocks: MAX_WAIT + 1, + preemptive_margin_blocks: 0, ..timing() }; - assert_eq!(cfg.danger_threshold(), 0); + let _ = cfg.danger_threshold(); + } + + #[test] + #[should_panic(expected = "nonzero seconds_per_block")] + fn wall_clock_helpers_fail_loud_on_zero_block_time_direct_construction() { + let cfg = ProtocolTiming { + seconds_per_block: 0, + ..timing() + }; + let _ = cfg.wall_clock_adjusted_danger_threshold(None, 0); } #[test] @@ -488,6 +716,25 @@ mod tests { ); } + #[test] + fn try_new_rejects_zero_seconds_per_block() { + assert_eq!( + ProtocolTiming::try_new(MAX_WAIT, 75, 1, 0), + Err(ProtocolTimingError::SecondsPerBlockZero), + ); + } + + #[test] + fn try_new_rejects_overflowing_read_staleness_window() { + assert_eq!( + ProtocolTiming::try_new(u64::MAX, 1, u64::MAX - 2, 2), + Err(ProtocolTimingError::ReadStaleWindowOverflow { + read_stale_after: u64::MAX - 2, + seconds_per_block: 2, + }), + ); + } + #[test] fn try_new_rejects_l1_read_stale_after_past_danger() { let danger_threshold = MAX_WAIT - 75; @@ -660,4 +907,10 @@ mod tests { assert_eq!(advance_expected_batch_nonce(0, vec![0, 2, 1]), 2); assert_eq!(advance_expected_batch_nonce(2, vec![2, 3]), 4); } + + #[test] + #[should_panic(expected = "expected batch nonce overflow: contract-impossible")] + fn advance_expected_batch_nonce_fails_loud_on_overflow() { + let _ = advance_expected_batch_nonce(u64::MAX, [u64::MAX]); + } } diff --git a/sequencer-core/src/scheduler/fold.rs b/sequencer-core/src/scheduler/fold.rs index 0b599a04..7de25107 100644 --- a/sequencer-core/src/scheduler/fold.rs +++ b/sequencer-core/src/scheduler/fold.rs @@ -23,7 +23,7 @@ use alloy_primitives::Address; use alloy_sol_types::Eip712Domain; use super::{Scheduler, SchedulerConfig, SchedulerInput}; -use crate::application::Application; +use crate::application::{AppError, Application}; /// One reconstructed L1 input for the fold, ordered ascending by inclusion /// block (ties broken by safe-input index at the source). Mirrors @@ -65,7 +65,7 @@ pub fn fold_replay( seeds: S, replay: R, stop_block: u64, -) -> (A, u64) +) -> Result<(A, u64), AppError> where A: Application, S: IntoIterator, @@ -122,18 +122,18 @@ where "fold replay inputs must arrive in ascending inclusion-block order" ); last_replay_block = Some(input.inclusion_block); - let _ = scheduler.process_input(SchedulerInput { + scheduler.process_input(SchedulerInput { sender: input.sender, inclusion_block: input.inclusion_block, domain: domain.clone(), payload: input.payload, - }); + })?; } // Step 5 — drain the leftover fridge at C. Every still-queued direct has // `inclusion_block <= C`, so this executes them all; the booting run, which // starts past C with no fridge, would otherwise never see them. - let _ = scheduler.drain_covered_at(stop_block); + scheduler.drain_covered_at(stop_block)?; // Fail loud on a dropped direct: after draining at C the fridge MUST be // empty. A non-empty fridge means an input arrived with `inclusion_block > C` @@ -147,13 +147,16 @@ where scheduler.queued_direct_len(), ); - scheduler.finish() + Ok(scheduler.finish()) } #[cfg(test)] mod tests { use super::*; - use crate::application::{AppError, AppOutputs, InvalidReason}; + use crate::application::{ + AppError, AppOutputs, ApplicationProgress, ApplyInputCapability, InvalidReason, + ProgressCommitCapability, + }; use crate::batch::{Batch, Frame}; use crate::l2_tx::{DirectInput, ValidUserOp}; use crate::user_op::UserOp; @@ -182,7 +185,8 @@ mod tests { #[derive(Default, Clone)] struct FoldApp { executed_directs: Vec, - safe_block: u64, + progress: ApplicationProgress, + fail_direct: Option, } impl Application for FoldApp { @@ -197,28 +201,39 @@ mod tests { Ok(()) } - fn execute_valid_user_op( + fn apply_valid_user_op( &mut self, + _capability: ApplyInputCapability<'_>, _user_op: &ValidUserOp, - safe_block: u64, + _safe_block: u64, ) -> Result { - self.safe_block = self.safe_block.max(safe_block); Ok(Vec::new()) } - fn execute_direct_input(&mut self, input: &DirectInput) -> Result { - self.executed_directs - .push(input.payload.first().copied().unwrap_or(0)); - self.safe_block = self.safe_block.max(input.block_number); + fn apply_direct_input( + &mut self, + _capability: ApplyInputCapability<'_>, + input: &DirectInput, + ) -> Result { + let marker = input.payload.first().copied().unwrap_or(0); + if self.fail_direct == Some(marker) { + return Err(AppError::Internal { + reason: format!("refusing direct {marker}"), + }); + } + self.executed_directs.push(marker); Ok(Vec::new()) } - fn executed_input_count(&self) -> u64 { - self.executed_directs.len() as u64 + fn execution_progress(&self) -> &ApplicationProgress { + &self.progress } - fn last_executed_safe_block(&self) -> u64 { - self.safe_block + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + &mut self.progress } fn from_dump(_prefix: &std::path::Path) -> Result { @@ -291,11 +306,37 @@ mod tests { Vec::new(), Vec::new(), 1_000, - ); + ) + .expect("empty fold"); assert_eq!(n, 7, "no batches ⇒ nonce stays at the checkpoint"); assert!(app.executed_directs.is_empty()); } + #[test] + fn fold_replay_propagates_application_error() { + let app = FoldApp { + fail_direct: Some(0xEE), + ..FoldApp::default() + }; + let error = match fold_replay( + app, + 0, + config(), + domain(), + vec![direct(DIRECT_SENDER, 10, 0xEE)], + Vec::new(), + 10, + ) { + Err(error) => error, + Ok(_) => panic!("application error must abort recovery fold"), + }; + + let AppError::Internal { reason } = error else { + panic!("unexpected application error: {error}"); + }; + assert_eq!(reason, "refusing direct 238"); + } + #[test] fn resume_nonce_advances_from_checkpoint_through_replayed_batches() { // Start at N0 = 5 (proves resume_at is honored, not 0); replay three @@ -313,7 +354,8 @@ mod tests { Vec::new(), replay, 1_000, - ); + ) + .expect("replay empty batches"); assert_eq!(n, 8, "three accepted batches advance N from 5 to 8"); } @@ -334,7 +376,8 @@ mod tests { seeds, replay, 1_000, - ); + ) + .expect("drain seeded directs"); assert_eq!( app.executed_directs, vec![0xAA, 0xBB], @@ -358,7 +401,8 @@ mod tests { seeds, Vec::new(), 30, - ); + ) + .expect("terminal drain"); assert_eq!( app.executed_directs, vec![0xCC], @@ -384,10 +428,13 @@ mod tests { 100, ) }; - let (app_a, n_a) = run(); - let (app_b, n_b) = run(); + let (app_a, n_a) = run().expect("first deterministic fold"); + let (app_b, n_b) = run().expect("second deterministic fold"); assert_eq!(app_a.executed_directs, app_b.executed_directs); - assert_eq!(app_a.safe_block, app_b.safe_block); + assert_eq!( + app_a.last_executed_safe_block(), + app_b.last_executed_safe_block() + ); assert_eq!(n_a, n_b); } @@ -531,15 +578,16 @@ mod tests { inclusion_block: input.inclusion_block, domain: domain(), payload: input.payload, - }); + })?; } + Ok::<(), AppError>(()) }; // LIVE: one scheduler over the whole stream, terminal-drained at C. let mut live = Scheduler::new(FoldApp::default(), config()); - feed(&mut live, pre_b()); - feed(&mut live, post_b()); - live.drain_covered_at(stop); + feed(&mut live, pre_b()).expect("live pre-checkpoint feed"); + feed(&mut live, post_b()).expect("live post-checkpoint feed"); + live.drain_covered_at(stop).expect("live terminal drain"); let (live_app, live_nonce) = live.finish(); // CHECKPOINT @ B: a scheduler over (genesis, B] then `finish` WITHOUT a @@ -547,7 +595,7 @@ mod tests { // as seeds. This is exactly the checkpoint contract: every batch ≤ B // applied, no (A,B] direct executed yet. let mut at_b = Scheduler::new(FoldApp::default(), config()); - feed(&mut at_b, pre_b()); + feed(&mut at_b, pre_b()).expect("checkpoint feed"); let (checkpoint_app, checkpoint_nonce) = at_b.finish(); let a = checkpoint_app.last_executed_safe_block(); assert!(a < 12 && 12 <= 15, "the seed direct(2) must sit in (A, B]"); @@ -562,7 +610,8 @@ mod tests { seeds, post_b(), stop, - ); + ) + .expect("recovery fold"); assert_eq!( recovered_nonce, live_nonce, diff --git a/sequencer-core/src/scheduler/mod.rs b/sequencer-core/src/scheduler/mod.rs index 4f1aa00c..cdff4422 100644 --- a/sequencer-core/src/scheduler/mod.rs +++ b/sequencer-core/src/scheduler/mod.rs @@ -5,8 +5,9 @@ pub mod fold; pub use fold::{FoldInput, fold_replay}; -use crate::application::{AppOutputs, Application}; +use crate::application::{AppError, AppOutputs, Application, ExecutionOutcome}; use crate::batch::{Batch, Frame, WireUserOp}; +use crate::history::ExecutedInputCount; use crate::l2_tx::DirectInput; use alloy_primitives::{Address, Signature}; use alloy_sol_types::Eip712Domain; @@ -69,12 +70,17 @@ impl ProcessResult { } } +// Test-assertion sugar only: asymmetric cross-type equality in a public +// consensus API makes `a == b` type-directed and non-obvious, so it stays out +// of the production surface. +#[cfg(test)] impl PartialEq for ProcessResult { fn eq(&self, other: &ProcessOutcome) -> bool { self.outcome == *other } } +#[cfg(test)] impl PartialEq for ProcessOutcome { fn eq(&self, other: &ProcessResult) -> bool { *self == other.outcome @@ -130,6 +136,11 @@ impl Scheduler { } pub fn new(app: A, config: SchedulerConfig) -> Self { + assert_eq!( + app.executed_input_count(), + ExecutedInputCount::ZERO, + "a genesis scheduler application must start at executed_input_count = 0" + ); Self::resume_at(app, config, 0) } @@ -166,10 +177,10 @@ impl Scheduler { /// still-queued direct has `inclusion_block <= C`, so this drains them all — /// exactly what the booting run's first frame at a safe block `>= C` would /// do, except the booting run (bare-metal, no fridge) never sees them. - pub fn drain_covered_at(&mut self, safe_block: u64) -> AppOutputs { + pub fn drain_covered_at(&mut self, safe_block: u64) -> Result { let mut outputs = Vec::new(); - self.drain_directs_safe_at(safe_block, &mut outputs); - outputs + self.drain_directs_safe_at(safe_block, &mut outputs)?; + Ok(outputs) } /// Consume the scheduler, returning the advanced application state `S'` and @@ -192,10 +203,10 @@ impl Scheduler { .map_err(|err| InspectError::Application(err.to_string())) } - pub fn process_input(&mut self, input: SchedulerInput) -> ProcessResult { + pub fn process_input(&mut self, input: SchedulerInput) -> Result { // Execute overdue directs before any input to keep backstop semantics explicit. let mut outputs = Vec::new(); - self.force_execute_overdue(input.inclusion_block, &mut outputs); + self.force_execute_overdue(input.inclusion_block, &mut outputs)?; if input.sender != self.config.sequencer_address { self.direct_q.push_back(QueuedDirectInput { @@ -203,12 +214,12 @@ impl Scheduler { payload: input.payload, inclusion_block: input.inclusion_block, }); - ProcessResult::new(ProcessOutcome::DirectEnqueued, outputs) + Ok(ProcessResult::new(ProcessOutcome::DirectEnqueued, outputs)) } else { let batch_result = - self.process_batch_payload(input.inclusion_block, &input.domain, &input.payload); + self.process_batch_payload(input.inclusion_block, &input.domain, &input.payload)?; outputs.extend(batch_result.outputs); - ProcessResult::new(batch_result.outcome, outputs) + Ok(ProcessResult::new(batch_result.outcome, outputs)) } } @@ -217,31 +228,36 @@ impl Scheduler { inclusion_block: u64, domain: &Eip712Domain, payload: &[u8], - ) -> ProcessResult { + ) -> Result { let Ok(batch): Result = ssz::Decode::from_ssz_bytes(payload) else { - return ProcessResult::without_outputs(ProcessOutcome::BatchRejected( - BatchRejectReason::DecodeFailed, + return Ok(ProcessResult::without_outputs( + ProcessOutcome::BatchRejected(BatchRejectReason::DecodeFailed), )); }; if batch.nonce != self.next_expected_batch_nonce { - return ProcessResult::without_outputs(ProcessOutcome::BatchRejected( - BatchRejectReason::WrongNonce { + return Ok(ProcessResult::without_outputs( + ProcessOutcome::BatchRejected(BatchRejectReason::WrongNonce { expected: self.next_expected_batch_nonce, got: batch.nonce, - }, + }), )); } let Some((frame_head, frame_tail)) = batch.frames.split_first() else { - self.advance_expected_batch_nonce(); - return ProcessResult::without_outputs(ProcessOutcome::BatchExecuted); + let next_nonce = self.checked_next_batch_nonce(); + self.next_expected_batch_nonce = next_nonce; + return Ok(ProcessResult::without_outputs( + ProcessOutcome::BatchExecuted, + )); }; if let Some(reason) = self.batch_reject_reason_for_block(inclusion_block, frame_head, frame_tail) { - return ProcessResult::without_outputs(ProcessOutcome::BatchRejected(reason)); + return Ok(ProcessResult::without_outputs( + ProcessOutcome::BatchRejected(reason), + )); } if has_elapsed_since( @@ -249,24 +265,28 @@ impl Scheduler { self.config.max_wait_blocks, inclusion_block, ) { - return ProcessResult::without_outputs(ProcessOutcome::BatchSkippedStale); + return Ok(ProcessResult::without_outputs( + ProcessOutcome::BatchSkippedStale, + )); } + // Preflight nonce exhaustion before any frame can mutate application + // state. A batch at `u64::MAX` has no canonical successor. + let next_nonce = self.checked_next_batch_nonce(); let mut outputs = Vec::new(); for frame in &batch.frames { - self.drain_directs_safe_at(frame.safe_block, &mut outputs); - self.execute_frame_user_ops(domain, frame, &mut outputs); + self.drain_directs_safe_at(frame.safe_block, &mut outputs)?; + self.execute_frame_user_ops(domain, frame, &mut outputs)?; } - self.advance_expected_batch_nonce(); - ProcessResult::new(ProcessOutcome::BatchExecuted, outputs) + self.next_expected_batch_nonce = next_nonce; + Ok(ProcessResult::new(ProcessOutcome::BatchExecuted, outputs)) } - fn advance_expected_batch_nonce(&mut self) { - self.next_expected_batch_nonce = self - .next_expected_batch_nonce + fn checked_next_batch_nonce(&self) -> u64 { + self.next_expected_batch_nonce .checked_add(1) - .expect("batch nonce overflow"); + .expect("batch nonce overflow: no canonical successor") } fn batch_reject_reason_for_block( @@ -302,7 +322,7 @@ impl Scheduler { domain: &Eip712Domain, frame: &Frame, outputs: &mut AppOutputs, - ) { + ) -> Result<(), AppError> { for user_op in &frame.user_ops { // An unrecoverable signature is dropped silently (the scheduler is a // pure deterministic fold; diagnostics would be a nondeterministic @@ -315,27 +335,13 @@ impl Scheduler { &plain, frame.fee_price, frame.safe_block, - ) { - Ok(crate::application::ExecutionOutcome::Included { - outputs: user_op_outputs, - }) => outputs.extend(user_op_outputs), - // Invalid op or app error: skip it (no state change, no output). - // - // The `Err(AppError)` arm is the canonical (fold) half of an - // asymmetry with the inclusion lane (`execute_user_op` in - // `inclusion_lane/mod.rs`), which fails *loud* on the same - // error. Duality (I1) still holds: an `Err` excludes the op - // from state on *both* sides (neither extends `outputs`), so - // the canonical state agrees — the lane merely additionally - // aborts, treating the error as the internal-invariant breach - // it is. Dead by construction today: the wallet app's - // `execute_valid_user_op` errors only on a fee/balance check - // it already passed in `validate_user_op`, which cannot change - // between the two calls within one fold step. - Ok(crate::application::ExecutionOutcome::Invalid(_)) | Err(_) => {} + )? { + ExecutionOutcome::Included(executed) => outputs.extend(executed.outputs), + ExecutionOutcome::Invalid(_) => {} } } } + Ok(()) } fn recover_sender(&self, domain: &Eip712Domain, wire_user_op: &WireUserOp) -> Option
{ @@ -348,25 +354,32 @@ impl Scheduler { signature.recover_address_from_prehash(&signing_hash).ok() } - fn drain_directs_safe_at(&mut self, safe_block: u64, outputs: &mut AppOutputs) { + fn drain_directs_safe_at( + &mut self, + safe_block: u64, + outputs: &mut AppOutputs, + ) -> Result<(), AppError> { while let Some(front) = self.direct_q.front() { if front.inclusion_block > safe_block { break; } - let queued = self.direct_q.pop_front().expect("queue front must exist"); let input = DirectInput { - sender: queued.sender, - block_number: queued.inclusion_block, - payload: queued.payload, + sender: front.sender, + block_number: front.inclusion_block, + payload: front.payload.clone(), }; - // A failing direct is skipped (deterministic fold; no diagnostics). - if let Ok(direct_outputs) = self.app.execute_direct_input(&input) { - outputs.extend(direct_outputs); - } + let executed = crate::application::execute_direct_input(&mut self.app, &input)?; + outputs.extend(executed.outputs); + self.direct_q.pop_front().expect("queue front must exist"); } + Ok(()) } - fn force_execute_overdue(&mut self, current_block: u64, outputs: &mut AppOutputs) { + fn force_execute_overdue( + &mut self, + current_block: u64, + outputs: &mut AppOutputs, + ) -> Result<(), AppError> { while let Some(front) = self.direct_q.front() { if has_elapsed_since( front.inclusion_block, @@ -378,21 +391,24 @@ impl Scheduler { block_number: front.inclusion_block, payload: front.payload.clone(), }; - // A failing overdue direct is skipped (deterministic fold). - if let Ok(direct_outputs) = self.app.execute_direct_input(&input) { - outputs.extend(direct_outputs); - } + let executed = crate::application::execute_direct_input(&mut self.app, &input)?; + outputs.extend(executed.outputs); self.direct_q.pop_front().expect("queue front must exist"); } else { break; } } + Ok(()) } } +/// Scheduler-local spelling of the one staleness predicate. Delegates to +/// [`crate::protocol::age_exceeds`] so the fold and the off-chain protocol +/// module cannot drift. Note the argument order differs: +/// `has_elapsed_since(start, wait, current) == age_exceeds(current, start, wait)`. fn has_elapsed_since(start_block: u64, wait_blocks: u64, current_block: u64) -> bool { - current_block.saturating_sub(start_block) >= wait_blocks + crate::protocol::age_exceeds(current_block, start_block, wait_blocks) } pub fn input_domain(chain_id: u64, verifying_contract: Address) -> Eip712Domain { @@ -402,6 +418,7 @@ pub fn input_domain(chain_id: u64, verifying_contract: Address) -> Eip712Domain #[cfg(test)] mod tests { use super::*; + use crate::application::{ApplicationProgress, ApplyInputCapability, ProgressCommitCapability}; use crate::user_op::UserOp; use alloy_primitives::{U256, address}; use k256::ecdsa::SigningKey; @@ -413,7 +430,8 @@ mod tests { executed: Vec, balances: std::collections::HashMap, nonces: std::collections::HashMap, - last_executed_safe_block: u64, + progress: ApplicationProgress, + fail_on: Option, } #[cfg(test)] @@ -442,6 +460,16 @@ mod tests { self.balances .insert(sender, current.saturating_add(U256::from(amount))); } + + fn with_progress(executed_input_count: u64, last_executed_safe_block: u64) -> Self { + Self { + progress: ApplicationProgress::new( + ExecutedInputCount::new(executed_input_count), + last_executed_safe_block, + ), + ..Self::default() + } + } } #[cfg(test)] @@ -481,12 +509,20 @@ mod tests { Ok(()) } - fn execute_valid_user_op( + fn apply_valid_user_op( &mut self, + _capability: ApplyInputCapability<'_>, user_op: &crate::l2_tx::ValidUserOp, - safe_block: u64, + _safe_block: u64, ) -> Result { - self.last_executed_safe_block = self.last_executed_safe_block.max(safe_block); + let marker = user_op.data.first().copied().unwrap_or_default(); + let event = RecordedTx::UserOp(marker); + if self.fail_on.as_ref() == Some(&event) { + return Err(AppError::Internal { + reason: format!("refusing {event:?}"), + }); + } + let sender = user_op.sender; let fee = crate::fee::fee_to_linear(user_op.fee); let balance = self.balance_of(sender); @@ -499,27 +535,35 @@ mod tests { let next_nonce = self.nonce_of(sender).wrapping_add(1); self.nonces.insert(sender, next_nonce); - let marker = user_op.data.first().copied().unwrap_or_default(); - self.executed.push(RecordedTx::UserOp(marker)); + self.executed.push(event); Ok(Vec::new()) } - fn execute_direct_input( + fn apply_direct_input( &mut self, + _capability: ApplyInputCapability<'_>, input: &DirectInput, ) -> Result { let marker = input.payload.first().copied().unwrap_or(0); - self.executed.push(RecordedTx::Direct(marker)); - self.last_executed_safe_block = self.last_executed_safe_block.max(input.block_number); + let event = RecordedTx::Direct(marker); + if self.fail_on.as_ref() == Some(&event) { + return Err(AppError::Internal { + reason: format!("refusing {event:?}"), + }); + } + self.executed.push(event); Ok(Vec::new()) } - fn executed_input_count(&self) -> u64 { - self.executed.len() as u64 + fn execution_progress(&self) -> &ApplicationProgress { + &self.progress } - fn last_executed_safe_block(&self) -> u64 { - self.last_executed_safe_block + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + &mut self.progress } fn from_dump(_prefix: &std::path::Path) -> Result { @@ -573,6 +617,15 @@ mod tests { } } + fn process_ok( + scheduler: &mut Scheduler, + input: SchedulerInput, + ) -> ProcessResult { + scheduler + .process_input(input) + .expect("test application execution must succeed") + } + fn address_from_signing_key(signing_key: &SigningKey) -> Address { let verifying = signing_key.verifying_key().to_encoded_point(false); Address::from_raw_public_key(&verifying.as_bytes()[1..]) @@ -627,7 +680,7 @@ mod tests { ); assert_eq!( - scheduler.process_input(direct_input(10, 1)), + process_ok(&mut scheduler, direct_input(10, 1)), ProcessOutcome::DirectEnqueued ); @@ -651,7 +704,7 @@ mod tests { }; assert_eq!( - scheduler.process_input(batch_input(20, batch)), + process_ok(&mut scheduler, batch_input(20, batch)), ProcessOutcome::BatchExecuted ); assert_eq!( @@ -671,7 +724,7 @@ mod tests { }, ); - scheduler.process_input(direct_input(1, 1)); + process_ok(&mut scheduler, direct_input(1, 1)); let signing_key = SigningKey::from_bytes((&[2_u8; 32]).into()).expect("signing key"); let sender = address_from_signing_key(&signing_key); scheduler.app.credit(sender, 1); @@ -690,7 +743,7 @@ mod tests { }], }; - scheduler.process_input(batch_input(6, batch)); + process_ok(&mut scheduler, batch_input(6, batch)); assert_eq!( scheduler.app.events(), [RecordedTx::Direct(1), RecordedTx::UserOp(2)] @@ -707,7 +760,7 @@ mod tests { }, ); - scheduler.process_input(direct_input(1, 9)); + process_ok(&mut scheduler, direct_input(1, 9)); let signing_key = SigningKey::from_bytes((&[3_u8; 32]).into()).expect("signing key"); let stale_batch = Batch { nonce: 0, @@ -724,7 +777,7 @@ mod tests { }], }; - let outcome = scheduler.process_input(batch_input(10, stale_batch)); + let outcome = process_ok(&mut scheduler, batch_input(10, stale_batch)); assert_eq!(outcome, ProcessOutcome::BatchSkippedStale); assert_eq!(scheduler.app.events(), [RecordedTx::Direct(9)]); // Stale batches do NOT consume the nonce — they are true no-ops in nonce space. @@ -751,7 +804,7 @@ mod tests { }; assert_eq!( - scheduler.process_input(batch_input(10, fresh_batch)), + process_ok(&mut scheduler, batch_input(10, fresh_batch)), ProcessOutcome::BatchExecuted ); } @@ -797,7 +850,7 @@ mod tests { }; assert_eq!( - scheduler.process_input(batch_input(10, invalid)), + process_ok(&mut scheduler, batch_input(10, invalid)), ProcessOutcome::BatchRejected(BatchRejectReason::NonMonotonicSafeBlocks) ); assert!(scheduler.app.events().is_empty()); @@ -831,7 +884,7 @@ mod tests { }; assert_eq!( - scheduler.process_input(batch_input(10, invalid)), + process_ok(&mut scheduler, batch_input(10, invalid)), ProcessOutcome::BatchRejected(BatchRejectReason::SafeBlockAboveInclusionBlock) ); assert!(scheduler.app.events().is_empty()); @@ -848,8 +901,8 @@ mod tests { }, ); - scheduler.process_input(direct_input(10, 1)); - scheduler.process_input(direct_input(11, 2)); + process_ok(&mut scheduler, direct_input(10, 1)); + process_ok(&mut scheduler, direct_input(11, 2)); let batch = Batch { nonce: 0, frames: vec![Frame { @@ -859,7 +912,7 @@ mod tests { }], }; - scheduler.process_input(batch_input(12, batch)); + process_ok(&mut scheduler, batch_input(12, batch)); assert_eq!(scheduler.app.events(), [RecordedTx::Direct(1)]); assert_eq!(scheduler.queued_direct_len(), 1); } @@ -881,13 +934,13 @@ mod tests { payload: vec![0xFF, 0xEE, 0xDD], }; assert_eq!( - scheduler.process_input(bad_batch), + process_ok(&mut scheduler, bad_batch), ProcessOutcome::BatchRejected(BatchRejectReason::DecodeFailed) ); assert_eq!(scheduler.next_expected_batch_nonce(), 0); assert_eq!( - scheduler.process_input(direct_input(11, 3)), + process_ok(&mut scheduler, direct_input(11, 3)), ProcessOutcome::DirectEnqueued ); assert_eq!(scheduler.queued_direct_len(), 1); @@ -903,9 +956,9 @@ mod tests { }, ); - scheduler.process_input(direct_input(1, 1)); - scheduler.process_input(direct_input(2, 2)); - scheduler.process_input(direct_input(8, 3)); + process_ok(&mut scheduler, direct_input(1, 1)); + process_ok(&mut scheduler, direct_input(2, 2)); + process_ok(&mut scheduler, direct_input(8, 3)); assert_eq!( scheduler.app.events(), @@ -939,7 +992,7 @@ mod tests { }; assert_eq!( - scheduler.process_input(batch_input(1, batch)), + process_ok(&mut scheduler, batch_input(1, batch)), ProcessOutcome::BatchExecuted ); assert!(scheduler.app.events().is_empty()); @@ -993,7 +1046,7 @@ mod tests { }; assert_eq!( - scheduler.process_input(batch_input(1, batch)), + process_ok(&mut scheduler, batch_input(1, batch)), ProcessOutcome::BatchExecuted ); assert_eq!(scheduler.app.events(), [RecordedTx::UserOp(4)]); @@ -1015,7 +1068,7 @@ mod tests { }; assert_eq!( - scheduler.process_input(batch_input(10, batch)), + process_ok(&mut scheduler, batch_input(10, batch)), ProcessOutcome::BatchExecuted ); assert!(scheduler.app.events().is_empty()); @@ -1061,7 +1114,7 @@ mod tests { }; assert_eq!( - scheduler.process_input(input), + process_ok(&mut scheduler, input), ProcessOutcome::BatchExecuted ); assert_eq!(scheduler.app.events(), [RecordedTx::UserOp(9)]); @@ -1077,7 +1130,7 @@ mod tests { }, ); assert_eq!( - scheduler.process_input(direct_input(1, 7)), + process_ok(&mut scheduler, direct_input(1, 7)), ProcessOutcome::DirectEnqueued ); // Inspect reflects executed app state, not the direct-input queue. @@ -1095,7 +1148,7 @@ mod tests { }], }; assert_eq!( - scheduler.process_input(batch_input(2, batch)), + process_ok(&mut scheduler, batch_input(2, batch)), ProcessOutcome::BatchExecuted ); @@ -1141,7 +1194,7 @@ mod tests { }; assert_eq!( - scheduler.process_input(batch_input(1, batch)), + process_ok(&mut scheduler, batch_input(1, batch)), ProcessOutcome::BatchRejected(BatchRejectReason::WrongNonce { expected: 0, got: 1, @@ -1161,6 +1214,164 @@ mod tests { // omission (it trusts the sequencer to emit well-formed batches), pinned // explicitly at the end so any *other* drift fails this test. + #[test] + #[should_panic( + expected = "a genesis scheduler application must start at executed_input_count = 0" + )] + fn genesis_scheduler_rejects_nonzero_application_progress() { + let _ = Scheduler::new( + RecordingApp::with_progress(1, 0), + SchedulerConfig { + sequencer_address: SEQUENCER, + max_wait_blocks: 100, + }, + ); + } + + #[test] + fn application_error_is_fatal_before_later_user_ops_and_nonce_commit() { + let app = RecordingApp { + fail_on: Some(RecordedTx::UserOp(1)), + ..RecordingApp::default() + }; + let mut scheduler = Scheduler::new( + app, + SchedulerConfig { + sequencer_address: SEQUENCER, + max_wait_blocks: 100, + }, + ); + + let signing_key = SigningKey::from_bytes((&[17_u8; 32]).into()).expect("signing key"); + let sender = address_from_signing_key(&signing_key); + scheduler.app.credit(sender, 2); + let batch = Batch { + nonce: 0, + frames: vec![Frame { + user_ops: vec![ + sign_wire_user_op(&test_domain(), &signing_key, 0, 1, vec![1]), + sign_wire_user_op(&test_domain(), &signing_key, 1, 1, vec![2]), + ], + safe_block: 1, + fee_price: 0, + }], + }; + + let error = scheduler + .process_input(batch_input(1, batch)) + .expect_err("application error must abort scheduler processing"); + let AppError::Internal { reason } = error else { + panic!("unexpected application error: {error}"); + }; + assert_eq!(reason, "refusing UserOp(1)"); + assert!(scheduler.app.events().is_empty()); + assert_eq!( + scheduler.app.executed_input_count(), + ExecutedInputCount::ZERO + ); + assert_eq!(scheduler.next_expected_batch_nonce(), 0); + } + + #[test] + fn overdue_direct_error_is_fatal_before_current_input_is_classified() { + let app = RecordingApp { + fail_on: Some(RecordedTx::Direct(1)), + ..RecordingApp::default() + }; + let mut scheduler = Scheduler::new( + app, + SchedulerConfig { + sequencer_address: SEQUENCER, + max_wait_blocks: 5, + }, + ); + + assert_eq!( + process_ok(&mut scheduler, direct_input(1, 1)), + ProcessOutcome::DirectEnqueued + ); + let error = scheduler + .process_input(direct_input(6, 2)) + .expect_err("overdue direct failure must abort scheduler processing"); + let AppError::Internal { reason } = error else { + panic!("unexpected application error: {error}"); + }; + assert_eq!(reason, "refusing Direct(1)"); + assert!(scheduler.app.events().is_empty()); + assert_eq!( + scheduler.app.executed_input_count(), + ExecutedInputCount::ZERO + ); + assert_eq!(scheduler.queued_direct_len(), 1); + } + + #[test] + fn batch_nonce_overflow_panics_before_application_mutation() { + let mut scheduler = Scheduler::resume_at( + RecordingApp::default(), + SchedulerConfig { + sequencer_address: SEQUENCER, + max_wait_blocks: 100, + }, + u64::MAX, + ); + let signing_key = SigningKey::from_bytes((&[18_u8; 32]).into()).expect("signing key"); + let sender = address_from_signing_key(&signing_key); + scheduler.app.credit(sender, 1); + let batch = Batch { + nonce: u64::MAX, + frames: vec![Frame { + user_ops: vec![sign_wire_user_op( + &test_domain(), + &signing_key, + 0, + 1, + vec![7], + )], + safe_block: 1, + fee_price: 0, + }], + }; + + let panic = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + let _ = scheduler.process_input(batch_input(1, batch)); + })); + assert!(panic.is_err(), "nonce exhaustion must fail loud"); + assert!(scheduler.app.events().is_empty()); + assert_eq!( + scheduler.app.executed_input_count(), + ExecutedInputCount::ZERO + ); + assert_eq!(scheduler.next_expected_batch_nonce(), u64::MAX); + } + + #[test] + fn executed_input_count_overflow_panics_before_application_mutation() { + let mut scheduler = Scheduler::resume_at( + RecordingApp::with_progress(u64::MAX, 0), + SchedulerConfig { + sequencer_address: SEQUENCER, + max_wait_blocks: 100, + }, + 0, + ); + assert_eq!( + process_ok(&mut scheduler, direct_input(1, 9)), + ProcessOutcome::DirectEnqueued + ); + + let panic = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + let _ = scheduler.drain_covered_at(1); + })); + assert!(panic.is_err(), "input-count exhaustion must fail loud"); + assert!(scheduler.app.events().is_empty()); + assert_eq!( + scheduler.app.executed_input_count(), + ExecutedInputCount::new(u64::MAX) + ); + assert_eq!(scheduler.queued_direct_len(), 1); + } + const DUALITY_MAX_WAIT: u64 = 5; fn duality_timing() -> crate::protocol::ProtocolTiming { @@ -1199,12 +1410,15 @@ mod tests { }, expected_nonce, ); - let canonical_executed = scheduler.process_input(SchedulerInput { - sender, - inclusion_block: inclusion, - domain: test_domain(), - payload: payload.to_vec(), - }) == ProcessOutcome::BatchExecuted; + let canonical_executed = process_ok( + &mut scheduler, + SchedulerInput { + sender, + inclusion_block: inclusion, + domain: test_domain(), + payload: payload.to_vec(), + }, + ) == ProcessOutcome::BatchExecuted; let offchain_accepted = duality_timing() .scheduler_accepts( SEQUENCER, @@ -1338,7 +1552,7 @@ mod tests { }, ); // A direct at block 1; by inclusion block 8 it is overdue (age 7 >= 5). - scheduler.process_input(direct_input(1, 1)); + process_ok(&mut scheduler, direct_input(1, 1)); let batch = if label == "wrong-nonce" { // expected 0, got 9 → BatchRejected(WrongNonce). @@ -1362,7 +1576,7 @@ mod tests { } }; - let outcome = scheduler.process_input(batch_input(8, batch)).outcome; + let outcome = process_ok(&mut scheduler, batch_input(8, batch)).outcome; assert!( matches!( outcome, @@ -1413,7 +1627,7 @@ mod tests { ], }; assert_eq!( - scheduler.process_input(batch_input(10, batch)), + process_ok(&mut scheduler, batch_input(10, batch)), ProcessOutcome::BatchRejected(BatchRejectReason::SafeBlockAboveInclusionBlock) ); assert_eq!(scheduler.next_expected_batch_nonce(), 0); diff --git a/sequencer/src/clock.rs b/sequencer/src/clock.rs new file mode 100644 index 00000000..fd22babf --- /dev/null +++ b/sequencer/src/clock.rs @@ -0,0 +1,42 @@ +// (c) Cartesi and individual authors (see AUTHORS) +// SPDX-License-Identifier: Apache-2.0 (see LICENSE) + +//! Shared clock helper. +//! +//! Every callsite that needs "now in Unix-ms" goes through [`unix_now_ms`] so +//! the sequencer has a single place to swap in a test clock if needed. +//! `SystemTime::now()` pre-epoch is defended against via `unwrap_or_default()`. + +use std::time::{Duration, SystemTime}; + +/// Current wall-clock time as Unix-ms. Passed into +/// [`crate::storage::Storage::check_danger`] and friends. +pub fn unix_now_ms() -> u64 { + let elapsed = SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default(); + duration_as_unix_ms(elapsed) +} + +fn duration_as_unix_ms(duration: Duration) -> u64 { + u64::try_from(duration.as_millis()).unwrap_or(u64::MAX) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn duration_to_unix_ms_saturates_before_narrowing() { + assert_eq!(duration_as_unix_ms(Duration::ZERO), 0); + assert_eq!( + duration_as_unix_ms(Duration::from_millis(u64::MAX)), + u64::MAX + ); + assert_eq!( + duration_as_unix_ms(Duration::new(u64::MAX, 999_999_999)), + u64::MAX, + "far-future duration must not wrap to a plausible small clock" + ); + } +} diff --git a/sequencer/src/runtime/config.rs b/sequencer/src/commands/config.rs similarity index 94% rename from sequencer/src/runtime/config.rs rename to sequencer/src/commands/config.rs index dd71360b..62fb24d6 100644 --- a/sequencer/src/runtime/config.rs +++ b/sequencer/src/commands/config.rs @@ -17,35 +17,14 @@ use alloy_primitives::Address; use clap::{ArgGroup, Args}; + +use crate::l1::SubmitterKey; use sequencer_core::protocol::{ProtocolTiming, ProtocolTimingError}; const DEFAULT_HTTP_ADDR: &str = "127.0.0.1:3000"; const DEFAULT_DATA_DIR: &str = "sequencer-data"; const DB_FILENAME: &str = "sequencer.db"; -/// Shared L1 / InputBox configuration used by both the input reader and the batch submitter. -/// -/// Built once at startup from the pinned deployment identity plus the runtime -/// `RunConfig`, so RPC URL, InputBox address, and app address are defined in a -/// single place and not duplicated across component configs. -#[derive(Debug, Clone)] -pub struct L1Config { - pub eth_rpc_url: String, - pub input_box_address: Address, - pub app_address: Address, - pub batch_submitter_private_key: String, - pub batch_submitter_address: Address, - /// The pinned deployment chain id. Carried here so keyed-write paths (e.g. - /// the preemptive-recovery flush) can re-confirm the RPC's chain id right - /// before signing via [`crate::l1::provider::create_verified_signer_provider`]. - pub chain_id: u64, - /// Opt into plaintext (`http://`) RPC against a non-loopback host — a - /// trusted private network (Docker/K8s service, private-VPC IP). Off by - /// default: the provider layer refuses remote plaintext otherwise. See - /// [`crate::l1::provider`]. - pub allow_insecure_rpc: bool, -} - /// Full path to the SQLite database file inside `data_dir`. pub fn db_path_in(data_dir: &str) -> String { std::path::Path::new(data_dir) @@ -231,9 +210,10 @@ pub struct KeyArgs { long, env = "CARTESI_SEQUENCER_AUTH_PRIVATE_KEY", hide_env_values = true, - group = "batch_submitter_key_source" + group = "batch_submitter_key_source", + value_parser = parse_submitter_key )] - batch_submitter_private_key: Option, + batch_submitter_private_key: Option, /// Path to a file whose first line contains the batch submitter private key. #[arg( long, @@ -246,7 +226,7 @@ pub struct KeyArgs { impl KeyArgs { /// Resolve the batch submitter private key from either the inline value or a key file. - pub fn resolve(&self) -> Result { + pub fn resolve(&self) -> Result { resolve_key_source( &self.batch_submitter_private_key, &self.batch_submitter_private_key_file, @@ -261,14 +241,14 @@ impl KeyArgs { /// "exactly one source" rule in code (the group can't be conditionally required /// at the clap level), so it needs the `None` case. fn resolve_key_source( - inline: &Option, + inline: &Option, file: &Option, -) -> Result, std::io::Error> { +) -> Result, std::io::Error> { if let Some(file) = file { let contents = std::fs::read_to_string(file)?; - Ok(Some( + Ok(Some(SubmitterKey::new( contents.lines().next().unwrap_or("").trim().to_string(), - )) + ))) } else { Ok(inline.clone()) } @@ -286,9 +266,10 @@ pub struct OptionalKeyArgs { #[arg( long, env = "CARTESI_SEQUENCER_AUTH_PRIVATE_KEY", - hide_env_values = true + hide_env_values = true, + value_parser = parse_submitter_key )] - batch_submitter_private_key: Option, + batch_submitter_private_key: Option, /// Path to a file whose first line is the batch-submitter private key. #[arg( long, @@ -312,7 +293,7 @@ impl OptionalKeyArgs { } /// Resolve the key if either source is set; `Ok(None)` when neither is. - fn resolve_if_present(&self) -> Result, std::io::Error> { + fn resolve_if_present(&self) -> Result, std::io::Error> { resolve_key_source( &self.batch_submitter_private_key, &self.batch_submitter_private_key_file, @@ -321,9 +302,9 @@ impl OptionalKeyArgs { } /// `setup` — establish the deployment's timeless state: pin identity, do the -/// initial L1 sync, register the genesis finalized snapshot, write the -/// setup-complete marker. L1-read-only: takes the batch-submitter address, not -/// the signing key. +/// initial L1 sync, register the genesis finalized snapshot, and atomically +/// complete setup. L1-read-only: takes the batch-submitter address, not the +/// signing key. #[derive(Debug, Clone, Args)] pub struct SetupConfig { #[arg(long, env = "CARTESI_SEQUENCER_DATA_DIR", default_value = DEFAULT_DATA_DIR, value_parser = parse_non_empty_string)] @@ -360,7 +341,7 @@ pub struct SetupConfig { /// bound of `setup`'s read-only detection scan: if a previous instance /// left any batch-submitter tx past `B` (or its wallet nonce is unsettled), /// `setup` refuses and points the operator at recovery. PR3 does not yet - /// load a non-genesis checkpoint machine — that is `setup --recovery` (PR5); + /// load a non-genesis checkpoint machine — that is `setup --recovery`; /// here `B` only scopes detection, so `B > 0` against a genesis-style setup /// merely narrows the scan. #[arg(long, env = "CARTESI_SEQUENCER_CHECKPOINT_BLOCK", default_value_t = 0)] @@ -441,7 +422,7 @@ impl SetupConfig { /// Resolve the recovery signing key. Precondition: [`SetupConfig::validate`] /// passed with `recovery == true` (so exactly one source is set). - pub fn resolve_recovery_key(&self) -> Result { + pub fn resolve_recovery_key(&self) -> Result { self.key .resolve_if_present() .map(|opt| opt.expect("recovery key presence is validated before resolve")) @@ -529,14 +510,14 @@ impl RunConfig { } /// Resolve the batch submitter private key from either the inline value or a key file. - pub fn resolve_private_key(&self) -> Result { + pub fn resolve_private_key(&self) -> Result { self.key.resolve() } } -/// `flush-mempool` — settle the batch-submitter wallet nonce on demand -///. Reads the submitter address + watermark from the DB; signs -/// no-op transactions, so it needs the key. +/// `flush-mempool` — settle the batch-submitter wallet nonce on demand. +/// Reads the submitter address + watermark from the DB and signs no-op +/// transactions, so it needs the key. #[derive(Debug, Clone, Args)] pub struct FlushConfig { #[arg(long, env = "CARTESI_SEQUENCER_DATA_DIR", default_value = DEFAULT_DATA_DIR, value_parser = parse_non_empty_string)] @@ -567,7 +548,7 @@ impl FlushConfig { db_path_in(&self.data_dir) } - pub fn resolve_private_key(&self) -> Result { + pub fn resolve_private_key(&self) -> Result { self.key.resolve() } } @@ -580,6 +561,12 @@ fn parse_non_empty_string(raw: &str) -> Result { Ok(value.to_string()) } +/// Wrap the inline key value at the clap edge so the secret is redacted the +/// moment it enters the process (env or flag). +fn parse_submitter_key(raw: &str) -> Result { + Ok(SubmitterKey::new(raw.to_string())) +} + fn parse_address(raw: &str) -> Result { if !raw.starts_with("0x") { return Err("address must be 0x-prefixed".to_string()); @@ -749,7 +736,12 @@ mod tests { assert!(cfg.recovery); assert_eq!(cfg.checkpoint_block, 1200); assert_eq!(cfg.checkpoint_dump_dir.as_deref(), Some("/tmp/ckpt")); - assert_eq!(cfg.resolve_recovery_key().expect("resolve key"), TEST_KEY); + assert_eq!( + cfg.resolve_recovery_key() + .expect("resolve key") + .expose_secret(), + TEST_KEY + ); } #[test] @@ -967,7 +959,7 @@ mod tests { let mut cmd = std::process::Command::new(&exe); cmd.args([ "--exact", - "runtime::config::tests::help_does_not_leak_private_key_value", + "commands::config::tests::help_does_not_leak_private_key_value", "--quiet", ]) .env("SEQUENCER_HELP_LEAK_OUT", &out_path) diff --git a/sequencer/src/commands/error.rs b/sequencer/src/commands/error.rs new file mode 100644 index 00000000..0ba1f5c7 --- /dev/null +++ b/sequencer/src/commands/error.rs @@ -0,0 +1,1391 @@ +// (c) Cartesi and individual authors (see AUTHORS) +// SPDX-License-Identifier: Apache-2.0 (see LICENSE) + +//! Command error taxonomy. Three groupings: +//! +//! - [`BootstrapError`] / [`IdentityError`]: everything that can go wrong +//! before runtime workers come up — config validation, deployment-identity +//! guards, startup recovery, initial DB open. +//! - [`WorkerExit`] + per-worker `*Exit`: how each runtime worker exited. +//! - [`CommandError`]: the top-level error every command returns (run, +//! setup, flush, acknowledge — it was misleadingly named `RunError` until +//! the 2026-08-19 homing pass), with generic [`std::io::Error`] / +//! [`rusqlite::Error`] catch-alls that are used widely enough not to nest. + +use thiserror::Error; + +use crate::ingress::inclusion_lane::{ + InclusionLaneError, dump_info::referenced_artifact_io_is_terminal, +}; +use crate::l1::fee_oracle::worker::FeeOracleError; +use crate::l1::reader::InputReaderError; +use crate::l1::submitter::BatchSubmitterError; +use crate::recovery::{DangerDetectorError, RecoveryError}; +use crate::storage::{ + DangerStatus, DeploymentIdentity, LifecycleError, StorageOpenError, + is_persistent_storage_error, is_persistent_storage_open_error, +}; +use sequencer_core::application::AppError; +use sequencer_core::protocol::ProtocolTimingError; + +// ── Top-level CommandError ──────────────────────────────────────────────── + +/// Top-level command error. Grouped by phase: +/// +/// - `Bootstrap`: startup failures before runtime workers come up. +/// - `Worker`: one of the runtime workers exited (server, inclusion lane, +/// input reader, batch submitter, danger detector, fee oracle). +/// - `Io` / `Storage`: generic catch-alls used widely; not worth nesting. +#[derive(Debug, Error)] +pub enum CommandError { + #[error("bootstrap failed: {0}")] + Bootstrap(#[from] BootstrapError), + #[error("worker exited: {0}")] + Worker(#[from] WorkerExit), + #[error("persistent storage invariant violation: {cause}")] + StorageInvariantViolation { cause: String }, + #[error("DB-referenced snapshot artifact {path:?} failed: {source}")] + ReferencedSnapshotArtifact { + path: std::path::PathBuf, + #[source] + source: std::io::Error, + }, + #[error(transparent)] + Io(#[from] std::io::Error), + #[error("storage operation failed: {0}")] + Storage(#[from] rusqlite::Error), + #[error(transparent)] + Lifecycle(#[from] LifecycleError), + #[error("application bootstrap failed: {0}")] + AppBootstrap(#[from] AppError), +} + +// ── Exit-code projection ──────────────────────────────────────────────── +// +// One exhaustive semantic verdict owns failure classification. The +// terminal-fault black box and the orchestrator-facing exit code both derive +// from it, so an operational projection can never become admission policy. +// The exit code remains an ops hint, never protocol authority over the next +// boot. Reserved: 1 (unclassified), 2 (clap usage), and 101 for a panic +// before the command harness can project trusted-code failures to 30. A +// terminal containment that cannot drain within the two-second watchdog +// bound exits via `abort()` (SIGABRT/134), bypassing this projection +// deliberately — supervisors must treat 134 from the sequencer as +// terminal-class (the cause is in the logs and, best-effort, the black box; +// a persistent fault re-detects fail-loud on the next boot that reads it). + +/// Restart with backoff; a recovery boot is expected next (it may take 15+ +/// min: flush + safe-finality wait). Startup probes must accommodate it. +pub const EXIT_RESTART_EXPECT_RECOVERY: u8 = 10; +/// Restart with backoff; a transient refusal that self-heals when the L1 view +/// freshens. Alert only if it persists. +pub const EXIT_RESTART_TRANSIENT: u8 = 20; +/// Terminal — do not restart; page an operator. The state cannot self-heal. +pub const EXIT_TERMINAL: u8 = 30; +/// Sticky — do **not** auto-restart-loop; an operator must wipe the +/// uncompleted data dir and run `setup --recovery`. The recovery sibling of +/// [`EXIT_RESTART_EXPECT_RECOVERY`] (10), but *operator-initiated*: 10 means +/// "restart and `run()` auto-recovers"; 40 means "a previous instance left work +/// past the checkpoint, and only an explicit fresh `setup --recovery` +/// can resolve it" — a plain restart of `setup` would re-detect and re-refuse +/// forever. Distinct from terminal (30) in that a known recovery procedure +/// *does* fix it. +pub const EXIT_SETUP_NEEDS_RECOVERY: u8 = 40; +/// Unclassified operational failure (for example, a provider error). Restart +/// with backoff. +pub const EXIT_UNCLASSIFIED: u8 = 1; + +/// Semantic disposition of a failed command. +/// +/// This is the single classification boundary shared by the terminal-fault +/// black box and the process exit-code projection: only [`Self::Terminal`] +/// best-effort records a cause (and exits 30: do not restart, page). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum CommandFailureVerdict { + ExpectedRecovery, + Retryable, + Terminal, + SetupRecoveryRequired, + Unclassified, +} + +impl CommandFailureVerdict { + /// Whether the black box records a terminal cause (and the process + /// exits 30: do not restart, page an operator). + pub(crate) const fn is_terminal(self) -> bool { + matches!(self, Self::Terminal) + } + + const fn exit_code(self) -> u8 { + match self { + Self::ExpectedRecovery => EXIT_RESTART_EXPECT_RECOVERY, + Self::Retryable => EXIT_RESTART_TRANSIENT, + Self::Terminal => EXIT_TERMINAL, + Self::SetupRecoveryRequired => EXIT_SETUP_NEEDS_RECOVERY, + Self::Unclassified => EXIT_UNCLASSIFIED, + } + } +} + +impl CommandError { + /// Classify this failure once for both the black-box recorder and the + /// supervisor-facing projection. + pub(crate) fn failure_verdict(&self) -> CommandFailureVerdict { + match self { + CommandError::Worker(WorkerExit::DangerDetected { status }) => { + danger_failure_verdict(status) + } + // A storage decoder panic means the durable state violated an + // internal contract. Restarting cannot repair that row. Egress + // reports the same condition through the supervisor's terminal + // fault signal; background tasks retain it in their exit shape. + CommandError::StorageInvariantViolation { .. } => CommandFailureVerdict::Terminal, + // Admission-fact refusals are terminal: the wrong command for + // this database, the absorbing divergence, a malformed black + // box. A plain storage failure underneath a lifecycle operation + // classifies by persistence like every other storage error — a + // transient SQLITE_BUSY on a black-box write must not page. + CommandError::Lifecycle( + LifecycleError::NotAdmissible { .. } + | LifecycleError::CanonicalDivergence { .. } + | LifecycleError::Malformed(_), + ) => CommandFailureVerdict::Terminal, + CommandError::Lifecycle(LifecycleError::Storage(source)) + if is_persistent_storage_error(source) => + { + CommandFailureVerdict::Terminal + } + CommandError::Lifecycle(LifecycleError::Storage(_)) => { + CommandFailureVerdict::Unclassified + } + CommandError::Storage(source) if is_persistent_storage_error(source) => { + CommandFailureVerdict::Terminal + } + CommandError::AppBootstrap(AppError::Internal { .. }) => { + CommandFailureVerdict::Terminal + } + CommandError::ReferencedSnapshotArtifact { source, .. } + if referenced_artifact_io_is_terminal(source) => + { + CommandFailureVerdict::Terminal + } + CommandError::Worker(exit) if exit.is_terminal() => CommandFailureVerdict::Terminal, + CommandError::Bootstrap(error) => bootstrap_failure_verdict(error), + // Worker crashes, provider errors, IO/storage/app catch-alls. + CommandError::Worker(_) + | CommandError::ReferencedSnapshotArtifact { .. } + | CommandError::Io(_) + | CommandError::Storage(_) + | CommandError::AppBootstrap(_) => CommandFailureVerdict::Unclassified, + } + } + + /// Project this error onto the exit-code contract. Clean shutdown + /// (exit 0) is handled by the caller — a `CommandError` is always a failure. + pub fn exit_code(&self) -> u8 { + self.failure_verdict().exit_code() + } +} + +// Per-worker terminality lives as `is_terminal_invariant` on each worker's +// own error type, beside its enum; `WorkerExit::is_terminal` above +// composes them. + +fn danger_failure_verdict(status: &DangerStatus) -> CommandFailureVerdict { + match status { + // A doomed closed batch / aging Tip: the next boot legitimately runs + // a slow recovery (flush + cascade). + DangerStatus::ClosedBatchInDanger(_) | DangerStatus::TipInDanger(_) => { + CommandFailureVerdict::ExpectedRecovery + } + // View-dependent refusals that self-heal once the provider recovers. + DangerStatus::L1ViewStale | DangerStatus::EstimatedBatchInDanger(_) => { + CommandFailureVerdict::Retryable + } + // The only genuinely terminal danger: canonical divergence. + DangerStatus::CanonicalDivergence(_) => CommandFailureVerdict::Terminal, + // `Safe` is never a detector exit. If it ever reaches here the danger + // classification is self-contradicting — page an operator (EXIT_TERMINAL) + // rather than silently restart-loop with backoff (EXIT_UNCLASSIFIED). + DangerStatus::Safe => { + debug_assert!( + false, + "danger_failure_verdict called with DangerStatus::Safe" + ); + CommandFailureVerdict::Terminal + } + } +} + +fn bootstrap_failure_verdict(err: &BootstrapError) -> CommandFailureVerdict { + match err { + BootstrapError::OpenStorage(source) if is_persistent_storage_open_error(source) => { + CommandFailureVerdict::Terminal + } + BootstrapError::Flush(source) if source.is_terminal_invariant() => { + CommandFailureVerdict::Terminal + } + // The recovery controller owns classification. No raw provider, + // storage, flush, or phase error crosses this boundary. + BootstrapError::Recovery(source) => { + if source.is_retryable() { + CommandFailureVerdict::Retryable + } else { + CommandFailureVerdict::Terminal + } + } + // Transient: self-heal when the L1 view / provider recovers — or, + // for the data-dir lock, when the previous owner finishes dying. + // `FeeOracleTransient` is documented "may self-heal", which is this + // class's definition. + BootstrapError::ChainIdRpc { .. } + | BootstrapError::Identity(IdentityError::FirstBootRequiresL1) + | BootstrapError::DetectionNonceRead { .. } + | BootstrapError::DataDirLocked { .. } + | BootstrapError::FeeOracleTransient { .. } + | BootstrapError::Flush(_) => CommandFailureVerdict::Retryable, + + // Terminal: needs an operator (wrong config, divergence, or a DB that + // was never set up). + BootstrapError::ChainIdMismatch { .. } + | BootstrapError::InvalidProtocolTiming(_) + | BootstrapError::FeeOracleMisconfig { .. } + | BootstrapError::FeeOracleFatal { .. } + | BootstrapError::SignerMisconfig { .. } + | BootstrapError::SetupNotComplete + | BootstrapError::CheckpointBeforeAppDeployment { .. } + | BootstrapError::SetupRecovery(_) + | BootstrapError::Identity(IdentityError::Mismatch { .. } | IdentityError::OrphanedState) => { + CommandFailureVerdict::Terminal + } + + // Sticky setup refusal: a previous instance left work past the + // checkpoint. Distinct from the auto-recovery class (10) — a plain + // restart re-refuses; only wipe + `setup --recovery` resolves it. + BootstrapError::SetupRefuse(_) => CommandFailureVerdict::SetupRecoveryRequired, + BootstrapError::OpenStorage(_) => CommandFailureVerdict::Unclassified, + } +} + +// ── Bootstrap-phase errors ───────────────────────────────────────────── + +/// Anything that can go wrong before runtime workers start: config validation, +/// deployment-identity guards, startup recovery, initial DB open. +#[derive(Debug, Error)] +pub enum BootstrapError { + #[error(transparent)] + OpenStorage(#[from] StorageOpenError), + #[error("RPC chain ID {rpc} does not match the expected chain ID {config}")] + ChainIdMismatch { rpc: u64, config: u64 }, + /// `eth_chainId` failed on a reachable RPC. We treat this as fatal + /// rather than warn-and-continue: proceeding with an unverified chain id + /// would pin a possibly-wrong deployment identity and poison subsequent + /// L1-unreachable boots, in addition to issuing soft confirmations + /// against the wrong chain's state. Operator should retry. + #[error("could not query chain ID from RPC: {message}")] + ChainIdRpc { message: String }, + /// Protocol-level config (`preemptive_margin_blocks` vs `max_wait_blocks`, + /// `l1_read_stale_after_blocks` vs `danger_threshold`) failed validation. + /// See [`ProtocolTimingError`]. + #[error(transparent)] + InvalidProtocolTiming(#[from] ProtocolTimingError), + /// Setup-pinned fee oracle configuration or validation is invalid. + #[error("fee oracle misconfiguration: {message}")] + FeeOracleMisconfig { message: String }, + /// A live quote/transport failed while bootstrapping. It may self-heal. + #[error("fee oracle bootstrap transient failure: {message}")] + FeeOracleTransient { message: String }, + /// Trusted-code join/arithmetic failures while setup persists the first + /// quote. These need operator attention but are not source misconfiguration. + #[error("fatal fee oracle bootstrap failure: {message}")] + FeeOracleFatal { message: String }, + /// The keyed signer provider could not be constructed (bad RPC URL or + /// private key). Deterministic operator misconfiguration: re-running the + /// same configuration re-fails identically, so it classifies terminal + /// like [`Self::ChainIdMismatch`] — the same semantic every command must + /// share (recovery, setup, and flush previously disagreed). + #[error("signer provider misconfiguration: {message}")] + SignerMisconfig { message: String }, + /// Startup recovery (or refusal) failed before runtime workers started. + #[error(transparent)] + Recovery(#[from] RecoveryError), + /// Deployment-identity guards — see [`IdentityError`]. + #[error(transparent)] + Identity(#[from] IdentityError), + /// `run` (or `flush-mempool`) was invoked against a DB where `setup` + /// has not completed — its completion fact is absent (setup never ran, or + /// crashed midway), or its outputs are incomplete. The + /// operator must run `setup` first; restarting `run` cannot self-heal. + #[error("setup has not completed for this data dir — run `setup` first")] + SetupNotComplete, + /// Another live process holds the exclusive data-directory lock + /// (`process_lock`). Retry-safe: the lock is what prevents concurrent + /// harm, and an orchestrated restart racing the previous owner's drain + /// resolves on its own. Two replicas pointed at one data dir show up as + /// this error repeating. + #[error( + "another process holds the data-directory lock ({path}); \ + refusing to run concurrently" + )] + DataDirLocked { path: String }, + /// The `flush-mempool` subcommand's flush failed (provider/transport). + #[error("mempool flush failed: {0}")] + Flush(#[from] crate::recovery::FlushError), + /// `setup`'s detection gate could not read the batch-submitter wallet + /// nonce from L1 (the one live RPC the gate makes). Transient — the + /// operator retries once the provider recovers; the prior sync already + /// proved L1 reachable, so this is a hiccup, not a misconfig. + #[error("setup detection: could not read submitter nonce from RPC: {message}")] + DetectionNonceRead { message: String }, + /// `setup`'s read-only detection gate found a previous instance left work + /// this checkpoint cannot account for. Sticky: only wiping the uncompleted + /// data dir and running `setup --recovery` resolves it — a plain + /// `setup` restart re-detects and re-refuses. + #[error(transparent)] + SetupRefuse(#[from] SetupRefuse), + /// `setup --checkpoint-block` predates the application's deployment block: + /// a promotion cannot have landed before the application contract existed. + /// Operator misconfig; restarting cannot self-heal. + #[error( + "checkpoint block {checkpoint_block} predates the application \ + deployment block {app_deployment_block}" + )] + CheckpointBeforeAppDeployment { + checkpoint_block: u64, + app_deployment_block: u64, + }, + /// `setup --recovery` failed in a way only the operator can fix (bad config, + /// a checkpoint that can't be loaded or doesn't fit the chain, or a DB that + /// is already set up). Terminal — see [`SetupRecoveryError`]. + #[error(transparent)] + SetupRecovery(#[from] SetupRecoveryError), +} + +/// Terminal failures of the `setup --recovery` procedure — the ones +/// an operator must resolve (the flush and the post-flush re-sync reuse the +/// transient [`RecoveryError`] paths instead). All map to [`EXIT_TERMINAL`]: +/// a plain restart re-runs the same bad inputs and re-fails identically. +#[derive(Debug, Error)] +pub enum SetupRecoveryError { + /// Cross-field config validation failed (recovery missing its dump dir / + /// checkpoint block / key, or a plain `setup` carrying recovery-only args). + /// See [`crate::commands::config::SetupConfig::validate`]. + #[error("invalid recovery configuration: {message}")] + InvalidConfig { message: String }, + /// `setup --recovery` was invoked against a DB that is already set up. + /// Recovery is a strict one-shot on a freshly-wiped DB (wipe and re-run with + /// `--recovery`); re-pointing a live deployment at a different checkpoint + /// would strand its existing state. + #[error( + "`setup --recovery` requires a freshly-wiped data dir, but this one is \ + already set up — wipe it and re-run" + )] + AlreadySetUp, + /// The checkpoint dump could not be loaded (missing/corrupt `info.toml`, or + /// the app's `from_dump` failed). Operator must supply a valid **sequencer** + /// dump dir (`info.toml` + `state/`), not a watchdog CM checkpoint. + #[error("failed to load checkpoint dump at {path}: {message}")] + CheckpointLoad { path: String, message: String }, + /// The checkpoint's last-executed safe block `A` is not strictly before the + /// checkpoint block `B`. The fold reconstructs the `(A, B]` fridge, so + /// `A < B` must hold — otherwise the checkpoint dump and + /// `--checkpoint-block` describe inconsistent points. + #[error( + "checkpoint last-executed safe block {executed_safe_block} (A) is not \ + before checkpoint block {checkpoint_block} (B)" + )] + CheckpointNotBeforeBlock { + executed_safe_block: u64, + checkpoint_block: u64, + }, + /// The input reader reported a successful post-flush sync without + /// persisting the safe-head observation that the recovery fold requires. + #[error( + "post-flush L1 resync completed without a persisted safe-head observation: \ + internal storage invariant violation" + )] + MissingResyncedSafeHead, + /// A re-run of `setup --recovery` found a root tip from a *prior* (crashed + /// before setup completion) attempt whose nonce differs from this + /// attempt's resume nonce — a different checkpoint, or the same one after the + /// post-flush head `C` advanced. The half-recovered DB cannot be resumed + /// onto a tree rooted at the old nonce (the anchor would move but the + /// existing root tip would not, silently breaking I16). Wipe the data dir + /// and re-run. + #[error( + "partial recovery: existing root tip carries nonce {existing_root_nonce}, \ + but this attempt resumes at {requested_nonce} — wipe the data dir and re-run" + )] + PartialRecoveryMismatch { + existing_root_nonce: u64, + requested_nonce: u64, + }, + /// A re-run of `setup --recovery` found a root tip carrying *this* attempt's + /// resume nonce but **no finalized snapshot** — a prior attempt that crashed + /// between opening the root tip and writing the snapshot. It cannot be + /// resumed safely: a re-sync may have advanced `C` with new direct inputs + /// (which leave `N'` unchanged) that resuming would leave unsequenced, so the + /// snapshot cursor would lag the folded `S'` and `run` would drain+execute + /// them a second time (divergence). Wipe the data dir and re-run (the + /// one-shot recovery model). + #[error( + "partial recovery: root tip at nonce {root_nonce} exists with no finalized \ + snapshot (crashed mid-fill) — wipe the data dir and re-run" + )] + PartialRecoveryIncomplete { root_nonce: u64 }, + /// `setup --recovery` found a finalized snapshot but **no root tip**. A + /// completed cockroach fill always has both (the tip is opened in step 2, + /// before the snapshot in step 4), so this is residue from a *different* + /// deployment mode left in the data dir — a plain `setup` that registered the + /// genesis finalized snapshot and crashed before setup completion. + /// Folding `(S', N')` and then silently keeping the old snapshot would mark + /// setup complete over the genesis state instead of the recovered state. Wipe + /// the data dir and re-run `setup --recovery`. + #[error( + "setup --recovery found a finalized snapshot (block {existing_finalized_block}) \ + with no root tip — residue from an incomplete plain `setup`; wipe the data \ + dir and re-run" + )] + RecoveryOverResidualSnapshot { existing_finalized_block: u64 }, + /// A plain (non-recovery) `setup` found a non-zero batch-tree anchor — + /// residue from a `setup --recovery` that crashed before completion. Booting + /// a genesis deployment over it would root the tree at the recovery nonce + /// instead of 0. Wipe the data dir, then run plain `setup` or re-run + /// `setup --recovery`. + #[error( + "plain setup found batch-tree anchor {anchor} (≠ 0) — leftover from an \ + incomplete `setup --recovery`; wipe the data dir and re-run" + )] + GenesisOverRecoveryResidue { anchor: u64 }, +} + +/// `setup`'s read-only detection gate: the reasons a +/// fresh `setup` refuses because a *previous* instance left work past the +/// checkpoint. Because plain setup has already initialized a genesis baseline, +/// the remedy is to wipe that uncompleted data dir and run `setup --recovery` +/// which flushes/folds the outstanding batches; a plain `setup` restart +/// re-detects and re-refuses (hence [`EXIT_SETUP_NEEDS_RECOVERY`], not the +/// auto-recovery class 10). +/// +/// Both variants carry diagnostic fields for the refusal log line. +#[derive(Debug, Error)] +pub enum SetupRefuse { + /// Step 1: the batch-submitter wallet nonce is not settled + /// (`pending > safe`) on the local provider — a previous instance left + /// pending or mined-but-unsafe batch txs. Local-view only: a + /// zombie tx dropped from this provider's pool but alive elsewhere evades + /// this check; bounded at runtime by the content-identity check. + #[error( + "batch-submitter wallet nonce not settled (pending {pending} > safe \ + {safe}) — a previous instance left in-flight batch txs; wipe this \ + uncompleted data dir, then run `setup --recovery`" + )] + WalletNonceUnsettled { pending: u64, safe: u64 }, + /// Step 2: a batch-submitter tx exists in `(checkpoint_block, safe]` — a + /// previous instance already wrote batches past this checkpoint, so a + /// genesis-style bootstrap would silently diverge from canonical state. + #[error( + "batch-submitter input found at block {found_block} past checkpoint \ + block {checkpoint_block} (safe_input_index {safe_input_index}) — wipe \ + this uncompleted data dir, then run `setup --recovery`" + )] + BatchPastCheckpoint { + checkpoint_block: u64, + found_block: u64, + safe_input_index: u64, + }, +} + +/// Deployment-identity failure modes. The sequencer pins itself to a specific +/// (chain_id, app_address, input_box_address, app_deployment_block, +/// batch_submitter_address, fee_oracle) tuple on first successful boot, then +/// refuses to run under a different identity to prevent silently associating +/// state from one deployment with another. +#[derive(Debug, Error)] +pub enum IdentityError { + /// L1 unreachable AND no cached identity in the DB. We need at least one + /// (live L1 query OR a prior boot's pinned identity) to safely bind this + /// sequencer to a deployment. Operator: bring up L1 and retry. + #[error("first boot requires L1: no cached deployment identity and L1 is unreachable")] + FirstBootRequiresL1, + /// The DB has persisted state but no pinned identity. Binding the current + /// config now would silently inherit an unknown deployment's data. + /// Operator: confirm provenance or wipe the DB. + #[error("orphaned state: DB has persisted state but no deployment identity to claim it")] + OrphanedState, + /// The pinned identity doesn't match the current config. + /// + /// `stored` and `expected` are boxed so the enum stays small — without + /// boxing this variant alone would push `CommandError`'s stack footprint past + /// 184 bytes, which clippy's `result_large_err` flags (and which inflates + /// every `Result<_, CommandError>` in the codebase, even successful returns). + /// The heap allocation is paid only on the error path, which is cold. + #[error("deployment identity mismatch ({fields}); stored={stored:?}; expected={expected:?}")] + Mismatch { + fields: String, + stored: Box, + expected: Box, + }, +} + +// ── Worker exits ─────────────────────────────────────────────────────── + +/// Which runtime worker exited, and why. One generic stop shape per worker +/// (this replaced six hand-copied per-worker enums), plus the danger +/// detector's deliberate `RecoveryRequired` trip as its own first-class arm: +/// not an error, but causes the runtime to exit so the orchestrator can +/// respawn into startup recovery. +#[derive(Debug, Error)] +pub enum WorkerExit { + #[error("server: {0}")] + Server(WorkerStop), + #[error("inclusion lane: {0}")] + Lane(WorkerStop), + #[error("input reader: {0}")] + InputReader(WorkerStop), + #[error("batch submitter: {0}")] + BatchSubmitter(WorkerStop), + #[error("danger detector: {0}")] + DangerDetector(WorkerStop), + #[error("fee oracle: {0}")] + FeeOracle(WorkerStop), + #[error("danger detector: danger detected ({status:?}) — stopping for startup recovery")] + DangerDetected { status: DangerStatus }, +} + +impl WorkerExit { + /// Whether this exit poisons the run (terminal, exit 30) rather than + /// restarting. Terminality is a method on each worker's own error type, + /// beside its enum — so adding a variant fails to compile there, not + /// silently classifying non-terminal through a distant wildcard. + /// An outer join failure is terminal exactly when it was a panic: + /// trusted sequencer/application code violated its execution contract + /// (fail-loud self-trust policy). + pub(crate) fn is_terminal(&self) -> bool { + match self { + // Listener/accept IO is environmental; restart. + WorkerExit::Server(stop) => stop.is_terminal_with(|_| false), + WorkerExit::Lane(stop) => { + stop.is_terminal_with(InclusionLaneError::is_terminal_invariant) + } + WorkerExit::InputReader(stop) => { + stop.is_terminal_with(InputReaderError::is_terminal_invariant) + } + WorkerExit::BatchSubmitter(stop) => { + stop.is_terminal_with(BatchSubmitterError::is_terminal_invariant) + } + WorkerExit::DangerDetector(stop) => { + stop.is_terminal_with(DangerDetectorError::is_terminal_invariant) + } + WorkerExit::FeeOracle(stop) => { + stop.is_terminal_with(FeeOracleError::is_terminal_invariant) + } + WorkerExit::DangerDetected { status } => { + matches!( + status, + DangerStatus::CanonicalDivergence(_) | DangerStatus::Safe + ) + } + } + } +} + +/// Generic worker stop shape: the task ended without runtime shutdown, ended +/// with its typed error, or failed to join. +#[derive(Debug, Error)] +pub enum WorkerStop { + #[error("stopped unexpectedly")] + StoppedUnexpectedly, + #[error("{0}")] + Source(E), + #[error("join error: {0}")] + Join(tokio::task::JoinError), +} + +impl WorkerStop { + /// Select-arm mapping: the runtime is live, so a clean `Ok(())` return + /// means the worker stopped on its own — unexpected. + pub(crate) fn from_select(result: Result, tokio::task::JoinError>) -> Self { + match result { + Ok(Ok(())) => Self::StoppedUnexpectedly, + Ok(Err(source)) => Self::Source(source), + Err(source) => Self::Join(source), + } + } + + /// Shutdown-path mapping: runtime-wide shutdown was already requested, + /// so `Ok(())` is the expected graceful drain. Distinct from + /// [`Self::from_select`], where the same `Ok(())` is unexpected. + pub(crate) fn from_shutdown( + result: Result, tokio::task::JoinError>, + ) -> Result<(), Self> { + match result { + Ok(Ok(())) => Ok(()), + Ok(Err(source)) => Err(Self::Source(source)), + Err(source) => Err(Self::Join(source)), + } + } + + fn is_terminal_with(&self, source_is_terminal: impl FnOnce(&E) -> bool) -> bool { + match self { + Self::StoppedUnexpectedly => false, + Self::Source(source) => source_is_terminal(source), + Self::Join(source) => source.is_panic(), + } + } +} + +// ── Chained `From` impls so `?` works at the top-level CommandError ──────── +// +// thiserror's `#[from]` is one-level; nested propagation needs manual +// impls. Each leaf error type that can bubble up through `?` in `run()` +// gets a direct From for CommandError. + +impl From for CommandError { + fn from(e: StorageOpenError) -> Self { + CommandError::Bootstrap(e.into()) + } +} + +/// The lock substrate keeps its own error type so `runtime/` never imports +/// this command-layer taxonomy; the classification (DataDirLocked is +/// retry-safe) lives here with its verdict. +impl From for CommandError { + fn from(e: crate::runtime::process_lock::ProcessLockError) -> Self { + use crate::runtime::process_lock::ProcessLockError; + match e { + ProcessLockError::Locked { path } => { + CommandError::Bootstrap(BootstrapError::DataDirLocked { path }) + } + ProcessLockError::Io(source) => CommandError::Io(source), + } + } +} + +/// One shared classification for the keyed signer-provider constructor, so +/// `setup`, `flush-mempool`, and any future keyed command cannot drift (the +/// three sites previously classified `Create` three different ways). The +/// run-recovery reducer keeps its own explicit Retry/Refuse polarity map — +/// that polarity is the reducer's to own — but its terminal/transient split +/// must agree with this one. +impl From for BootstrapError { + fn from(e: crate::l1::provider::VerifiedSignerProviderError) -> Self { + use crate::l1::provider::VerifiedSignerProviderError as E; + match e { + E::ChainIdMismatch { rpc, expected } => BootstrapError::ChainIdMismatch { + rpc, + config: expected, + }, + E::ChainIdRpc(message) => BootstrapError::ChainIdRpc { message }, + E::Create(message) => BootstrapError::SignerMisconfig { message }, + } + } +} + +impl From for CommandError { + fn from(e: crate::l1::provider::VerifiedSignerProviderError) -> Self { + CommandError::Bootstrap(e.into()) + } +} + +impl From for CommandError { + fn from(e: ProtocolTimingError) -> Self { + CommandError::Bootstrap(e.into()) + } +} + +impl From for CommandError { + fn from(e: RecoveryError) -> Self { + CommandError::Bootstrap(e.into()) + } +} + +impl From for CommandError { + fn from(e: IdentityError) -> Self { + CommandError::Bootstrap(e.into()) + } +} + +impl From for CommandError { + fn from(e: crate::recovery::FlushError) -> Self { + CommandError::Bootstrap(BootstrapError::Flush(e)) + } +} + +impl From for CommandError { + fn from(e: SetupRefuse) -> Self { + CommandError::Bootstrap(BootstrapError::SetupRefuse(e)) + } +} + +impl From for CommandError { + fn from(e: SetupRecoveryError) -> Self { + CommandError::Bootstrap(BootstrapError::SetupRecovery(e)) + } +} + +impl From for CommandError { + fn from(error: FeeOracleError) -> Self { + match error { + FeeOracleError::OpenStorage(error) => CommandError::from(error), + FeeOracleError::Storage(error) => CommandError::Storage(error), + FeeOracleError::Transient(message) => { + CommandError::Bootstrap(BootstrapError::FeeOracleTransient { message }) + } + // Named arm so the message isn't double-prefixed through the + // variant's own Display ("fee-oracle misconfiguration: ..."). + FeeOracleError::Misconfig(message) => { + CommandError::Bootstrap(BootstrapError::FeeOracleMisconfig { message }) + } + FeeOracleError::Join(message) => { + CommandError::Bootstrap(BootstrapError::FeeOracleFatal { + message: FeeOracleError::Join(message).to_string(), + }) + } + FeeOracleError::FatalMath(error) => { + CommandError::Bootstrap(BootstrapError::FeeOracleFatal { + message: FeeOracleError::FatalMath(error).to_string(), + }) + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::l1::fee_oracle::worker::FeeOracleError; + use crate::l1::submitter::BatchPosterError; + use crate::l1::watermark::WalletNonceWatermarkError; + use crate::recovery::{ + FlushError, RecoveryError, RecoveryFailure, RecoveryRefusalReason, RecoveryRetryReason, + }; + use crate::storage::DeploymentIdentity; + use sequencer_core::protocol::ProtocolTimingError; + + fn danger(status: DangerStatus) -> CommandError { + CommandError::Worker(WorkerExit::DangerDetected { status }) + } + + fn dummy_identity() -> DeploymentIdentity { + use alloy_primitives::Address; + DeploymentIdentity { + chain_id: 1, + app_address: Address::repeat_byte(0x11), + input_box_address: Address::repeat_byte(0x22), + app_deployment_block: 0, + batch_submitter_address: Address::repeat_byte(0x33), + fee_oracle: crate::storage::FeeOracleIdentity::Fixed { log_gas_price: 0 }, + } + } + + #[test] + fn semantic_verdict_drives_lifecycle_and_exit_projection() { + let cases = [ + ( + danger(DangerStatus::TipInDanger(3)), + CommandFailureVerdict::ExpectedRecovery, + EXIT_RESTART_EXPECT_RECOVERY, + ), + ( + CommandError::Bootstrap(BootstrapError::ChainIdRpc { + message: "provider unavailable".into(), + }), + CommandFailureVerdict::Retryable, + EXIT_RESTART_TRANSIENT, + ), + ( + CommandError::StorageInvariantViolation { + cause: "broken durable invariant".into(), + }, + CommandFailureVerdict::Terminal, + EXIT_TERMINAL, + ), + ( + CommandError::from(SetupRefuse::WalletNonceUnsettled { + pending: 14, + safe: 13, + }), + CommandFailureVerdict::SetupRecoveryRequired, + EXIT_SETUP_NEEDS_RECOVERY, + ), + ( + CommandError::Io(std::io::Error::other("operational failure")), + CommandFailureVerdict::Unclassified, + EXIT_UNCLASSIFIED, + ), + ]; + + for (error, expected_verdict, expected_exit_code) in cases { + let verdict = error.failure_verdict(); + assert_eq!(verdict, expected_verdict); + assert_eq!(error.exit_code(), expected_exit_code); + assert_eq!( + verdict.is_terminal(), + matches!(verdict, CommandFailureVerdict::Terminal) + ); + } + } + + #[test] + fn r4_class_10_expect_recovery_boot() { + assert_eq!( + danger(DangerStatus::ClosedBatchInDanger(0)).exit_code(), + EXIT_RESTART_EXPECT_RECOVERY + ); + assert_eq!( + danger(DangerStatus::TipInDanger(3)).exit_code(), + EXIT_RESTART_EXPECT_RECOVERY + ); + } + + #[test] + fn r4_class_20_transient_refusal() { + assert_eq!( + danger(DangerStatus::L1ViewStale).exit_code(), + EXIT_RESTART_TRANSIENT + ); + assert_eq!( + danger(DangerStatus::EstimatedBatchInDanger(2)).exit_code(), + EXIT_RESTART_TRANSIENT + ); + assert_eq!( + CommandError::Bootstrap(BootstrapError::Recovery(RecoveryError::retry( + RecoveryRetryReason::L1ViewStale, + ))) + .exit_code(), + EXIT_RESTART_TRANSIENT + ); + assert_eq!( + CommandError::Bootstrap(BootstrapError::Identity(IdentityError::FirstBootRequiresL1)) + .exit_code(), + EXIT_RESTART_TRANSIENT + ); + assert_eq!( + CommandError::Bootstrap(BootstrapError::ChainIdRpc { + message: "x".into() + }) + .exit_code(), + EXIT_RESTART_TRANSIENT + ); + assert_eq!( + CommandError::from(FlushError::Provider("x".into())).exit_code(), + EXIT_RESTART_TRANSIENT + ); + // Documented "may self-heal" — the definition of this class (it + // previously projected to unclassified/1, misleading restart policy). + assert_eq!( + CommandError::from(FeeOracleError::Transient("RPC unavailable".into())).exit_code(), + EXIT_RESTART_TRANSIENT + ); + assert_eq!( + CommandError::from(FlushError::Watermark(WalletNonceWatermarkError::Storage( + rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code: rusqlite::ffi::ErrorCode::DatabaseBusy, + extended_code: 5, + }, + None, + ) + ),)) + .exit_code(), + EXIT_RESTART_TRANSIENT, + "operational watermark contention remains retryable" + ); + // A flush failure surfaced via startup recovery must land in the same + // class as the flush-mempool subcommand's FlushError (review M2). + assert_eq!( + CommandError::Bootstrap(BootstrapError::Recovery(RecoveryError::retry( + RecoveryFailure::Flush(FlushError::Provider("x".into())), + ))) + .exit_code(), + EXIT_RESTART_TRANSIENT + ); + assert_eq!( + CommandError::Bootstrap(BootstrapError::Recovery(RecoveryError::retry( + RecoveryRetryReason::ResyncBehindFlushView { + resynced_safe_block: 1, + flush_observed_safe_block: 2, + }, + ))) + .exit_code(), + EXIT_RESTART_TRANSIENT + ); + } + + #[test] + fn r4_class_30_terminal_operator_required() { + assert_eq!( + CommandError::StorageInvariantViolation { + cause: "test cause".into() + } + .exit_code(), + EXIT_TERMINAL + ); + // A deterministic signer-construction misconfig (bad RPC URL or + // private key) classifies terminal in every command, matching the + // ChainIdMismatch precedent; setup/flush previously projected it + // unclassified while recovery refused it. + assert_eq!( + CommandError::from(crate::l1::provider::VerifiedSignerProviderError::Create( + "bad key".into() + )) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::ReferencedSnapshotArtifact { + path: "/durable/snapshot".into(), + source: std::io::Error::from(std::io::ErrorKind::NotFound), + } + .exit_code(), + EXIT_TERMINAL, + "a missing DB-referenced snapshot cannot self-heal on restart" + ); + assert_eq!( + CommandError::Storage(rusqlite::Error::QueryReturnedNoRows).exit_code(), + EXIT_TERMINAL, + "a mandatory durable row disappearing cannot self-heal on restart" + ); + assert_eq!( + CommandError::Worker(WorkerExit::DangerDetector(WorkerStop::Source( + DangerDetectorError::Storage(rusqlite::Error::QueryReturnedNoRows,) + ),)) + .exit_code(), + EXIT_TERMINAL, + "typed persistent storage errors retain terminal classification through workers" + ); + assert_eq!( + CommandError::Worker(WorkerExit::FeeOracle(WorkerStop::Source( + FeeOracleError::Storage(rusqlite::Error::QueryReturnedNoRows), + ))) + .exit_code(), + EXIT_TERMINAL, + "the newer fee-oracle worker retains persistent storage classification" + ); + assert_eq!( + CommandError::Worker(WorkerExit::FeeOracle(WorkerStop::Source( + FeeOracleError::Join("blocking storage task panicked".into()), + ))) + .exit_code(), + EXIT_TERMINAL, + "a fee-oracle blocking-task panic is a trusted-code failure" + ); + assert_eq!( + CommandError::Worker(WorkerExit::InputReader(WorkerStop::Source( + InputReaderError::StorageTaskPanicked { + operation: "reading corrupt state", + }, + ))) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Worker(WorkerExit::BatchSubmitter(WorkerStop::Source( + BatchSubmitterError::StorageTaskPanicked { + operation: "reading corrupt state", + }, + ))) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Worker(WorkerExit::DangerDetector(WorkerStop::Source( + DangerDetectorError::StorageTaskPanicked, + ))) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Worker(WorkerExit::BatchSubmitter(WorkerStop::Source( + BatchSubmitterError::Poster(BatchPosterError::StorageInvariantViolation), + ))) + .exit_code(), + EXIT_TERMINAL + ); + let persistent_watermark = + || WalletNonceWatermarkError::Storage(rusqlite::Error::QueryReturnedNoRows); + assert_eq!( + CommandError::Worker(WorkerExit::BatchSubmitter(WorkerStop::Source( + BatchSubmitterError::Poster(BatchPosterError::Watermark(persistent_watermark())), + ))) + .exit_code(), + EXIT_TERMINAL, + "persistent write-before-broadcast storage failure must not retry as a provider error" + ); + assert_eq!( + CommandError::from(FlushError::Watermark(persistent_watermark())).exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Bootstrap(BootstrapError::Recovery(RecoveryError::refuse( + RecoveryFailure::Flush(FlushError::Watermark(persistent_watermark())), + ))) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + danger(DangerStatus::CanonicalDivergence(0)).exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Bootstrap(BootstrapError::Recovery(RecoveryError::refuse( + RecoveryRefusalReason::CanonicalDivergence { nonce: 0 }, + ))) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Bootstrap(BootstrapError::SetupNotComplete).exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Bootstrap(BootstrapError::ChainIdMismatch { rpc: 1, config: 2 }) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Bootstrap(BootstrapError::InvalidProtocolTiming( + ProtocolTimingError::MarginNotLessThanMaxWait { + margin: 1200, + max_wait: 1200 + } + )) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Bootstrap(BootstrapError::Identity(IdentityError::OrphanedState)) + .exit_code(), + EXIT_TERMINAL + ); + // `setup --recovery` operator-fixable failures are terminal (a restart + // re-runs the same bad inputs). + assert_eq!( + CommandError::from(SetupRecoveryError::AlreadySetUp).exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Bootstrap(BootstrapError::Identity(IdentityError::Mismatch { + fields: "chain_id".into(), + stored: Box::new(dummy_identity()), + expected: Box::new(dummy_identity()), + })) + .exit_code(), + EXIT_TERMINAL + ); + // Partial-recovery residue: operator must wipe — terminal. + assert_eq!( + CommandError::from(SetupRecoveryError::PartialRecoveryMismatch { + existing_root_nonce: 3, + requested_nonce: 5, + }) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::from(SetupRecoveryError::GenesisOverRecoveryResidue { anchor: 7 }) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::from(SetupRecoveryError::PartialRecoveryIncomplete { root_nonce: 3 }) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::from(SetupRecoveryError::RecoveryOverResidualSnapshot { + existing_finalized_block: 0, + }) + .exit_code(), + EXIT_TERMINAL + ); + // Reader-level chain-id mismatch (warm-boot backstop) is terminal, like + // the boot-time BootstrapError::ChainIdMismatch. + assert_eq!( + CommandError::Worker(WorkerExit::InputReader(WorkerStop::Source( + InputReaderError::ChainIdMismatch { + rpc: 1, + expected: 31337, + } + ))) + .exit_code(), + EXIT_TERMINAL + ); + // The same mismatch surfacing during a startup-recovery safe-head sync + // (RecoveryError path) is terminal too — not the unclassified Recovery + // catch-all (which would loop on the wrong chain). + assert_eq!( + CommandError::Bootstrap(BootstrapError::Recovery(RecoveryError::refuse( + RecoveryFailure::InputReader(InputReaderError::ChainIdMismatch { + rpc: 1, + expected: 31337, + }), + ))) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Bootstrap(BootstrapError::Recovery(RecoveryError::refuse( + RecoveryFailure::InputReader(InputReaderError::StorageTaskPanicked { + operation: "startup sync", + }), + ))) + .exit_code(), + EXIT_TERMINAL + ); + } + + #[test] + fn startup_recovery_reader_persistent_open_failure_is_terminal() { + let source = StorageOpenError::Sqlite(rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code: rusqlite::ffi::ErrorCode::NotADatabase, + extended_code: 26, + }, + None, + )); + let error = CommandError::Bootstrap(BootstrapError::Recovery(RecoveryError::refuse( + RecoveryFailure::OpenStorage(source), + ))); + + assert_eq!(error.exit_code(), EXIT_TERMINAL); + } + + #[test] + fn startup_recovery_reader_busy_open_failure_remains_restartable() { + let source = StorageOpenError::Sqlite(rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code: rusqlite::ffi::ErrorCode::DatabaseBusy, + extended_code: 5, + }, + None, + )); + let error = CommandError::Bootstrap(BootstrapError::Recovery(RecoveryError::retry( + RecoveryFailure::OpenStorage(source), + ))); + + assert_eq!(error.exit_code(), EXIT_RESTART_TRANSIENT); + } + + #[test] + fn r4_class_40_setup_needs_operator_recovery() { + // Sticky setup refusals: the operator must wipe the uncompleted data + // dir and run `setup --recovery`, not plain-restart (which would + // re-detect and re-refuse) — so they get a dedicated code, distinct + // from the auto-recovery class (10). + assert_eq!( + CommandError::from(SetupRefuse::WalletNonceUnsettled { + pending: 14, + safe: 13, + }) + .exit_code(), + EXIT_SETUP_NEEDS_RECOVERY + ); + assert_eq!( + CommandError::from(SetupRefuse::BatchPastCheckpoint { + checkpoint_block: 100, + found_block: 250, + safe_input_index: 7, + }) + .exit_code(), + EXIT_SETUP_NEEDS_RECOVERY + ); + // A checkpoint predating genesis is operator misconfig — terminal (30), + // not a recovery trigger. + assert_eq!( + CommandError::Bootstrap(BootstrapError::CheckpointBeforeAppDeployment { + checkpoint_block: 5, + app_deployment_block: 10, + }) + .exit_code(), + EXIT_TERMINAL + ); + } + + #[test] + fn r4_class_1_unclassified() { + assert_eq!( + CommandError::Io(std::io::Error::other("boom")).exit_code(), + EXIT_UNCLASSIFIED + ); + assert_eq!( + CommandError::ReferencedSnapshotArtifact { + path: "/durable/snapshot".into(), + source: std::io::Error::other("filesystem unavailable"), + } + .exit_code(), + EXIT_UNCLASSIFIED, + "operational filesystem failures remain restartable" + ); + assert_eq!( + CommandError::Storage(rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code: rusqlite::ffi::ErrorCode::DatabaseBusy, + extended_code: 5, + }, + None, + )) + .exit_code(), + EXIT_UNCLASSIFIED, + "transient storage contention remains restartable" + ); + // Lifecycle classifies by variant, not wholesale: fact refusals + // page; a busy black-box write must not. + assert_eq!( + CommandError::Lifecycle(LifecycleError::Storage(rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code: rusqlite::ffi::ErrorCode::DatabaseBusy, + extended_code: 5, + }, + None, + ))) + .exit_code(), + EXIT_UNCLASSIFIED, + "a transient lifecycle storage failure remains restartable" + ); + assert_eq!( + CommandError::Lifecycle(LifecycleError::Storage( + rusqlite::Error::QueryReturnedNoRows + )) + .exit_code(), + EXIT_TERMINAL, + "a persistent lifecycle storage failure pages" + ); + assert_eq!( + CommandError::Lifecycle(LifecycleError::NotAdmissible { + requested: crate::storage::LifecycleCommand::Run, + reason: "setup has not completed for this data directory", + }) + .exit_code(), + EXIT_TERMINAL, + "an admission-fact refusal pages" + ); + assert_eq!( + CommandError::Lifecycle(LifecycleError::CanonicalDivergence { nonce: 7 }).exit_code(), + EXIT_TERMINAL, + "divergence pages" + ); + assert_eq!( + CommandError::Worker(WorkerExit::Server(WorkerStop::StoppedUnexpectedly)).exit_code(), + EXIT_UNCLASSIFIED + ); + assert_eq!( + CommandError::Worker(WorkerExit::FeeOracle(WorkerStop::Source( + FeeOracleError::Transient("RPC unavailable".into()), + ))) + .exit_code(), + EXIT_UNCLASSIFIED + ); + assert_eq!( + CommandError::Worker(WorkerExit::FeeOracle(WorkerStop::Source( + FeeOracleError::Storage(rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code: rusqlite::ffi::ErrorCode::DatabaseBusy, + extended_code: 5, + }, + None, + )), + ))) + .exit_code(), + EXIT_UNCLASSIFIED, + "fee-oracle storage contention remains restartable" + ); + } + + #[test] + fn fee_oracle_bootstrap_preserves_operational_storage_classification() { + for code in [ + rusqlite::ffi::ErrorCode::DatabaseBusy, + rusqlite::ffi::ErrorCode::DatabaseLocked, + ] { + let error = FeeOracleError::Storage(rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code, + extended_code: 5, + }, + None, + )); + let mapped = CommandError::from(error); + assert!(matches!(&mapped, CommandError::Storage(_))); + assert_eq!( + mapped.exit_code(), + EXIT_UNCLASSIFIED, + "SQLite contention remains restartable during setup's first quote" + ); + } + } + + #[test] + fn fee_oracle_fatal_math_is_terminal_on_bootstrap_and_worker() { + use crate::l1::fee_oracle::math::MathError; + assert_eq!( + CommandError::from(FeeOracleError::FatalMath(MathError::Overflow)).exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Worker(WorkerExit::FeeOracle(WorkerStop::Source( + FeeOracleError::FatalMath(MathError::Overflow), + ))) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Worker(WorkerExit::FeeOracle(WorkerStop::Source( + FeeOracleError::FatalMath(MathError::ExceedsRepresentableRange), + ))) + .exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::from(FeeOracleError::Misconfig("wrong pair".into())).exit_code(), + EXIT_TERMINAL + ); + assert_eq!( + CommandError::Worker(WorkerExit::FeeOracle(WorkerStop::Source( + FeeOracleError::Misconfig("wrong pair".into()), + ))) + .exit_code(), + EXIT_TERMINAL + ); + } + + #[test] + fn fee_oracle_shutdown_ok_is_graceful() { + let result: Result, tokio::task::JoinError> = Ok(Ok(())); + assert!(WorkerStop::from_shutdown(result).is_ok()); + } + + #[test] + fn r4_app_bootstrap_internal_error_is_terminal() { + assert_eq!( + CommandError::AppBootstrap(AppError::Internal { + reason: "application invariant failed".into(), + }) + .exit_code(), + EXIT_TERMINAL + ); + } + + #[test] + fn r4_app_bootstrap_io_error_is_unclassified() { + assert_eq!( + CommandError::AppBootstrap(AppError::Io(std::io::Error::other("disk unavailable"))) + .exit_code(), + EXIT_UNCLASSIFIED + ); + } + + #[tokio::test] + async fn r4_panicking_outer_worker_join_is_terminal() { + let source = tokio::spawn(async { + panic!("worker invariant failure"); + }) + .await + .expect_err("task must panic"); + assert_eq!( + CommandError::Worker(WorkerExit::Lane(WorkerStop::Join(source))).exit_code(), + EXIT_TERMINAL + ); + } +} diff --git a/sequencer/src/runtime/flush.rs b/sequencer/src/commands/flush.rs similarity index 57% rename from sequencer/src/runtime/flush.rs rename to sequencer/src/commands/flush.rs index c5094aaf..c4bfadb2 100644 --- a/sequencer/src/runtime/flush.rs +++ b/sequencer/src/commands/flush.rs @@ -9,32 +9,52 @@ //! the same flush `setup --recovery` will run internally. It is a keyed L1 //! write, so it needs the signing key; it reads the submitter address and the //! wallet-nonce watermark from the DB, so it refuses unless `setup` completed. -//! Flush-only — it does not cascade (that stays in preemptive recovery). +//! Flush-only — it does not sync or cascade (those stay in normal-run +//! recovery), and it requires a completed setup. A successful wallet flush +//! proves nothing about the rest of the runtime and is never treated as if +//! it did. -use super::config::FlushConfig; -use super::{BootstrapError, RunError, load_setup_identity}; -use crate::l1::provider::VerifiedSignerProviderError; +use super::load_setup_identity; +use crate::commands::config::FlushConfig; +use crate::commands::error::CommandError; use crate::recovery::MempoolFlusher; -use crate::storage; +use crate::storage::{self, LifecycleCommand}; -pub async fn flush_mempool(config: FlushConfig) -> Result<(), RunError> { +pub async fn flush_mempool(config: FlushConfig) -> Result<(), CommandError> { + std::fs::create_dir_all(&config.data_dir)?; + // Exclusive process ownership: a flush must never broadcast beside a + // live sequencer (or another flush) reading the same watermark. + let _process_lock = crate::runtime::process_lock::ProcessLock::acquire(&config.data_dir)?; let db_path = config.db_path(); - // Gate on a completed setup and read the pinned submitter address. + super::preflight_lifecycle_command(&db_path, LifecycleCommand::MaintenanceFlush)?; let identity = load_setup_identity(&db_path)?; // The signing key must match the pinned submitter — flushing under the // wrong key would settle the wrong account's nonce. let key = super::verify_submitter_key(config.resolve_private_key()?, &identity)?; - // The durable flush anchor (review R1a): every slot we ever broadcast + let result = flush_mempool_admitted(config, identity, key).await; + // Verdict-neutral black-box settlement. + super::record_terminal_fault_best_effort(&db_path, LifecycleCommand::MaintenanceFlush, &result); + result +} + +async fn flush_mempool_admitted( + config: FlushConfig, + identity: storage::DeploymentIdentity, + key: crate::l1::SubmitterKey, +) -> Result<(), CommandError> { + let db_path = config.db_path(); + + // The durable flush anchor: every slot we ever broadcast // must resolve at safe depth, regardless of the local pool's memory. let watermark = { - let mut storage = storage::Storage::open(&db_path)?; + let mut storage = storage::Storage::open_writer(&db_path)?; storage.wallet_nonce_watermark()? }; - // Wrong-chain RPC guard (review F6): flush broadcasts keyed L1 txs, so — + // Wrong-chain RPC guard: flush broadcasts keyed L1 txs, so — // like `setup` and `run` — it must confirm the RPC's chain id matches the // pinned one before signing, or it would burn submitter nonce slots on the // wrong chain. `create_verified_signer_provider` folds that check into the @@ -43,23 +63,12 @@ pub async fn flush_mempool(config: FlushConfig) -> Result<(), RunError> { // reachable anyway). let provider = crate::l1::provider::create_verified_signer_provider( &config.eth_rpc_url, - &key, + key.expose_secret(), identity.chain_id, config.allow_insecure_rpc, ) .await - .map_err(|e| match e { - VerifiedSignerProviderError::ChainIdMismatch { rpc, expected } => { - RunError::Bootstrap(BootstrapError::ChainIdMismatch { - rpc, - config: expected, - }) - } - VerifiedSignerProviderError::ChainIdRpc(message) => { - RunError::Bootstrap(BootstrapError::ChainIdRpc { message }) - } - VerifiedSignerProviderError::Create(msg) => RunError::Io(std::io::Error::other(msg)), - })?; + .map_err(CommandError::from)?; let safe_block = MempoolFlusher::flush_to_safe( provider, identity.batch_submitter_address, diff --git a/sequencer/src/commands/mod.rs b/sequencer/src/commands/mod.rs new file mode 100644 index 00000000..5c87018b --- /dev/null +++ b/sequencer/src/commands/mod.rs @@ -0,0 +1,336 @@ +// (c) Cartesi and individual authors (see AUTHORS) +// SPDX-License-Identifier: Apache-2.0 (see LICENSE) + +//! The operator commands: each child module is one command *bracket* — +//! acquire the exclusive process lock, preflight the admission facts, +//! execute the body, and best-effort record a terminal cause in the black +//! box. The *mechanisms* the brackets invoke live in their domain modules +//! (`crate::recovery` for the reducer/flusher, `crate::runtime` for the +//! shared authority machinery: process lock, `RuntimeScope`). +//! +//! - [`run`] — boot workers from a set-up DB (plus `workers`, its supervisor) +//! - [`setup`] — establish the timeless deployment state +//! - [`flush`] — settle the wallet nonce without launching +//! +//! This module hosts the helpers shared by more than one bracket (identity +//! gates, keyed-writer verification, lifecycle preflights). The CLI harness +//! that dispatches the commands lives in [`crate::harness`]. + +pub mod config; +pub mod error; +pub mod flush; +pub mod run; +pub mod setup; +#[cfg(test)] +pub(crate) mod test_support; + +pub use error::{ + BootstrapError, CommandError, IdentityError, SetupRecoveryError, SetupRefuse, WorkerExit, +}; + +use std::time::Duration; + +use crate::storage::{self, DeploymentIdentity, LifecycleCommand}; +use alloy_primitives::Address; + +pub(crate) const INPUT_READER_POLL_INTERVAL: Duration = Duration::from_secs(2); + +fn preflight_lifecycle_command( + db_path: &str, + command: LifecycleCommand, +) -> Result<(), CommandError> { + if !std::path::Path::new(db_path).try_exists()? { + return Err(BootstrapError::SetupNotComplete.into()); + } + let storage = storage::Storage::open_read_only(db_path)?; + storage.preflight_lifecycle_command(command)?; + Ok(()) +} + +/// Verdict-neutral black-box settlement: when the command ended terminal, +/// record its cause best-effort. Telemetry must never change a verdict — a +/// failed record loses only the black-box copy, and the exit code and logs +/// still carry it. +pub(crate) fn record_terminal_fault_best_effort( + db_path: &str, + command: LifecycleCommand, + result: &Result<(), CommandError>, +) { + let Err(error) = result else { return }; + if !error.failure_verdict().is_terminal() { + return; + } + let cause = error.to_string(); + let recorded = storage::Storage::open_writer(db_path) + .map_err(|open_error| open_error.to_string()) + .and_then(|mut storage| { + storage + .record_terminal_fault(command, &cause) + .map_err(|record_error| record_error.to_string()) + }); + if let Err(record_error) = recorded { + tracing::warn!( + error = %record_error, + cause = %cause, + "terminal cause not recorded in the black box; the exit code and this log carry the verdict" + ); + } +} + +pub(crate) fn batch_submitter_address_from_private_key( + private_key: &str, +) -> Result { + use alloy::signers::local::PrivateKeySigner; + use std::str::FromStr; + + // Deterministic operator misconfig — terminal, like every signer + // misconfiguration. The message never echoes key material. + Ok(PrivateKeySigner::from_str(private_key) + .map_err(|_| { + CommandError::Bootstrap(BootstrapError::SignerMisconfig { + message: "invalid batch submitter private key".to_string(), + }) + })? + .address()) +} + +/// Gate `run`/`flush` on a completed `setup` and return the pinned identity. +/// A missing completion fact — or a completion fact without an identity (a +/// corrupt/incomplete setup) — is a terminal `SetupNotComplete`: the operator must +/// (re-)run `setup`, not retry `run`. +pub(crate) fn load_setup_identity(db_path: &str) -> Result { + let storage = storage::Storage::open_read_only(db_path)?; + if !storage.is_setup_complete()? { + return Err(BootstrapError::SetupNotComplete.into()); + } + match storage.deployment_identity()? { + Some(identity) => Ok(identity), + None => Err(BootstrapError::SetupNotComplete.into()), + } +} + +/// Verify that the RPC's `eth_chainId` matches the configured chain id. +/// +/// Treated as fatal on mismatch *and* on RPC error: pinning a wrong or +/// unverified chain id into storage would poison subsequent L1-unreachable +/// boots and issue soft confirmations against the wrong chain. Caller is +/// expected to retry on `ChainIdRpc`. +pub(crate) async fn validate_rpc_chain_id( + eth_rpc_url: &str, + expected: u64, + allow_insecure: bool, +) -> Result<(), CommandError> { + use alloy::providers::Provider; + let check_provider = crate::l1::provider::create_provider(eth_rpc_url, allow_insecure) + .map_err(|e| CommandError::Io(std::io::Error::other(e)))?; + match check_provider.get_chain_id().await { + Ok(rpc_chain_id) if rpc_chain_id != expected => { + Err(CommandError::Bootstrap(BootstrapError::ChainIdMismatch { + rpc: rpc_chain_id, + config: expected, + })) + } + Ok(_) => Ok(()), + Err(e) => Err(CommandError::Bootstrap(BootstrapError::ChainIdRpc { + message: e.to_string(), + })), + } +} + +pub(crate) fn ensure_deployment_identity( + db_path: &str, + expected: DeploymentIdentity, +) -> Result<(), CommandError> { + let mut storage = storage::Storage::open(db_path)?; + if let Some(stored) = storage.deployment_identity()? { + return require_deployment_identity_match(stored, expected); + } + if storage.has_persisted_deployment_state()? { + return Err(IdentityError::OrphanedState.into()); + } + let stored = storage.load_or_insert_deployment_identity(expected)?; + require_deployment_identity_match(stored, expected) +} + +fn require_deployment_identity_match( + stored: DeploymentIdentity, + expected: DeploymentIdentity, +) -> Result<(), CommandError> { + let fields = deployment_identity_mismatch_fields(stored, expected); + if fields.is_empty() { + return Ok(()); + } + Err(IdentityError::Mismatch { + fields: fields.join(", "), + stored: Box::new(stored), + expected: Box::new(expected), + } + .into()) +} + +/// Keyed-writer preflight shared by `run` and `flush`: confirm a resolved +/// batch-submitter signing `key` signs for the submitter `setup` pinned in +/// `identity`, returning the key on success. Both subcommands broadcast keyed +/// L1 txs, so signing under the wrong key would consume the wrong wallet's +/// nonce slots — a fail-loud identity mismatch, not a recoverable condition. +pub(crate) fn verify_submitter_key( + key: crate::l1::SubmitterKey, + identity: &DeploymentIdentity, +) -> Result { + let key_address = batch_submitter_address_from_private_key(key.expose_secret())?; + if key_address != identity.batch_submitter_address { + let expected = DeploymentIdentity { + batch_submitter_address: key_address, + ..*identity + }; + require_deployment_identity_match(*identity, expected)?; + } + Ok(key) +} + +fn deployment_identity_mismatch_fields( + stored: DeploymentIdentity, + expected: DeploymentIdentity, +) -> Vec<&'static str> { + let mut fields = Vec::new(); + if stored.chain_id != expected.chain_id { + fields.push("chain_id"); + } + if stored.app_address != expected.app_address { + fields.push("app_address"); + } + if stored.input_box_address != expected.input_box_address { + fields.push("input_box_address"); + } + if stored.app_deployment_block != expected.app_deployment_block { + fields.push("app_deployment_block"); + } + if stored.batch_submitter_address != expected.batch_submitter_address { + fields.push("batch_submitter_address"); + } + if stored.fee_oracle != expected.fee_oracle { + fields.push("fee_oracle"); + } + fields +} + +#[cfg(test)] +mod tests { + use super::{ + BootstrapError, CommandError, IdentityError, batch_submitter_address_from_private_key, + deployment_identity_mismatch_fields, ensure_deployment_identity, + require_deployment_identity_match, + }; + use crate::recovery::{RecoveryError, RecoveryRetryReason}; + use crate::storage::test_helpers::{SENDER_A, default_protocol_timing, temp_db}; + use crate::storage::{DeploymentIdentity, Storage}; + use alloy_primitives::Address; + use sequencer_core::protocol::ProtocolTimingError; + + // Margin/stale-boundary validation is exercised directly in + // `sequencer-core/src/protocol.rs`. The runtime tests below only cover + // the typed `From` conversions into `CommandError` and the bootstrap-time + // identity guards. Worker `From` conversions live in + // `run::workers`. + + #[test] + fn invalid_protocol_config_propagates_through_run_error() { + let err: CommandError = ProtocolTimingError::MarginNotLessThanMaxWait { + margin: 1200, + max_wait: 1200, + } + .into(); + assert!(matches!( + err, + CommandError::Bootstrap(BootstrapError::InvalidProtocolTiming(_)) + )); + } + + #[test] + fn startup_recovery_error_preserves_recovery_category() { + let err: CommandError = RecoveryError::retry(RecoveryRetryReason::L1ViewStale).into(); + assert!(matches!( + err, + CommandError::Bootstrap(BootstrapError::Recovery(RecoveryError::Retry(_))) + )); + } + + fn identity() -> DeploymentIdentity { + DeploymentIdentity { + chain_id: 31337, + app_address: Address::repeat_byte(0x11), + input_box_address: Address::repeat_byte(0x22), + app_deployment_block: 42, + batch_submitter_address: Address::repeat_byte(0x33), + fee_oracle: crate::storage::FeeOracleIdentity::Fixed { log_gas_price: 0 }, + } + } + + #[test] + fn deployment_identity_match_accepts_same_identity() { + let identity = identity(); + require_deployment_identity_match(identity, identity).expect("same identity should match"); + } + + #[test] + fn deployment_identity_mismatch_reports_changed_fields() { + let stored = identity(); + let expected = DeploymentIdentity { + chain_id: 31338, + app_address: Address::repeat_byte(0x44), + batch_submitter_address: Address::repeat_byte(0x55), + ..stored + }; + + assert_eq!( + deployment_identity_mismatch_fields(stored, expected), + vec!["chain_id", "app_address", "batch_submitter_address"] + ); + let err = require_deployment_identity_match(stored, expected) + .expect_err("mismatch should refuse startup"); + assert!(matches!( + err, + CommandError::Bootstrap(BootstrapError::Identity(IdentityError::Mismatch { fields, .. })) + if fields == "chain_id, app_address, batch_submitter_address" + )); + } + + #[test] + fn deployment_identity_refuses_non_empty_unpinned_db() { + let db = temp_db("runtime-unpinned-deployment-state"); + { + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + storage + .append_safe_inputs(0, &[], SENDER_A, &default_protocol_timing()) + .expect("seed deployment-bound state"); + } + + let err = ensure_deployment_identity(db.path.as_str(), identity()) + .expect_err("non-empty unpinned DB must refuse"); + assert!(matches!( + err, + CommandError::Bootstrap(BootstrapError::Identity(IdentityError::OrphanedState)) + )); + } + + #[test] + fn invalid_private_key_is_terminal_misconfig_and_does_not_echo_key_material() { + let secret = "0xabc123SECRET"; + let err = batch_submitter_address_from_private_key(secret) + .expect_err("invalid private key should be rejected"); + let message = err.to_string(); + + assert!( + matches!( + err, + CommandError::Bootstrap(BootstrapError::SignerMisconfig { .. }) + ), + "a malformed key is deterministic operator misconfig (terminal), got {err:?}" + ); + assert_eq!(err.exit_code(), crate::commands::error::EXIT_TERMINAL); + assert!( + !message.contains(secret), + "private key material must not be reflected in startup errors" + ); + } +} diff --git a/sequencer/src/commands/run/mod.rs b/sequencer/src/commands/run/mod.rs new file mode 100644 index 00000000..50f2e79c --- /dev/null +++ b/sequencer/src/commands/run/mod.rs @@ -0,0 +1,247 @@ +// (c) Cartesi and individual authors (see AUTHORS) +// SPDX-License-Identifier: Apache-2.0 (see LICENSE) + +//! The `run` command: boot the sequencer from an already-set-up DB. +//! +//! Phases: +//! +//! 1. **Gate + identity**: refuse unless `setup` completed; read the pinned +//! deployment identity from the DB (chain id / app address are no longer +//! CLI args — they come from the identity). +//! 2. **Recovery reducer**: inspect one local fact set, execute at most one +//! phase, and re-inspect until the reducer selects admission +//! (`crate::recovery` owns the mechanism; this bracket only invokes it). +//! 3. **Prepare + admit + launch**: prepare every fallible runtime resource, +//! re-run the reducer over one consistent fact set, then consume the +//! single-use admission in one non-yielding worker launch (`workers`). +//! +//! Two senses of "admitted": phase 1 is the *lifecycle* gate (the +//! `_admitted` suffix all three command brackets share); the *runtime* +//! admission witness is minted later, in phase 3. + +mod startup_hygiene; +mod workers; + +use std::time::Duration; + +use crate::commands::config::RunConfig; +use crate::commands::error::CommandError; +use crate::commands::{ + INPUT_READER_POLL_INTERVAL, load_setup_identity, preflight_lifecycle_command, + record_terminal_fault_best_effort, verify_submitter_key, +}; +use crate::l1::L1Config; +use crate::l1::reader::{InputReader, InputReaderConfig}; +use crate::runtime::process_lock; +use crate::storage::{self, LifecycleCommand}; +use sequencer_core::application::Application; + +use workers::{PreparedRuntime, WorkersConfig}; + +/// Boot the sequencer from an already-set-up DB. Generic over the app type +/// (for the lane's `from_dump`, the egress state-file path, and the +/// max-payload bound) but takes no app *value* — `setup` already registered +/// the genesis snapshot, so the lane reloads via `A::from_dump`. +pub async fn run(config: RunConfig) -> Result<(), CommandError> +where + A: Application + Clone + Sync + 'static, +{ + // ── Gate + identity ────────────────────────────────────── + std::fs::create_dir_all(&config.data_dir)?; + // Exclusive process ownership, ahead of every read. The controller keeps + // its own clone through durable settlement; runtime-owned tasks separately + // retain clones until they stop. + let process_lock = process_lock::ProcessLock::acquire(&config.data_dir)?; + let db_path = config.db_path(); + let timing = config.protocol_timing()?; + + // Local absorbing facts are inspected before identity/key checks and any + // RPC: canonical divergence and the two-sided setup-completion rule. + // Divergence is never reinterpreted as a provider failure. + preflight_lifecycle_command(&db_path, LifecycleCommand::Run)?; + let identity = load_setup_identity(&db_path)?; + + // `run` holds the signing key (it submits). The key's address must match + // the pinned submitter address — running with the wrong key against a DB + // pinned to another submitter is a fail-loud identity mismatch. + let key = verify_submitter_key(config.resolve_private_key()?, &identity)?; + + // The identity travels verbatim inside the L1 bundle from here on, so + // exactly one route to the pinned values exists below this gate. + let l1_config = L1Config { + identity, + eth_rpc_url: config.eth_rpc_url.clone(), + batch_submitter_private_key: key, + allow_insecure_rpc: config.allow_insecure_rpc, + }; + + let result = run_admitted::(config, timing, l1_config, process_lock.clone()).await; + // The Ok-path divergence fact check (run's counterpart of the one + // `complete_setup` keeps): divergence persisted during this run must + // exit terminal even through a clean drain — exit 0 is the one code + // that breaks the supervisor's rediscovery chain (restart → preflight + // refusal), and the detector's poll cadence leaves a window where a + // clean shutdown outruns re-detection. + let result = result.and_then(|()| refuse_divergence_on_clean_exit(&db_path)); + // Verdict-neutral black-box settlement: a terminal failure records + // its cause best-effort — never changing the verdict — while a + // panic/cancellation/SIGKILL writes nothing; the next boot proceeds and + // re-derives everything from facts. + record_terminal_fault_best_effort(&db_path, LifecycleCommand::Run, &result); + result +} + +fn refuse_divergence_on_clean_exit(db_path: &str) -> Result<(), CommandError> { + let mut storage = storage::Storage::open_read_only(db_path)?; + if let Some((nonce, _)) = storage.canonical_divergence()? { + return Err(storage::LifecycleError::CanonicalDivergence { nonce }.into()); + } + Ok(()) +} + +async fn run_admitted( + config: RunConfig, + timing: sequencer_core::protocol::ProtocolTiming, + l1_config: L1Config, + process_lock: process_lock::ProcessLock, +) -> Result<(), CommandError> +where + A: Application + Clone + Sync + 'static, +{ + let db_path = config.db_path(); + + // `run` never re-discovers identity from L1 — it builds the reader from + // the pinned InputBox address + app deployment block and syncs incrementally. + let mut input_reader = InputReader::from_parts( + InputReaderConfig { + rpc_url: config.eth_rpc_url.clone(), + allow_insecure_rpc: config.allow_insecure_rpc, + app_address: l1_config.identity.app_address, + poll_interval: INPUT_READER_POLL_INTERVAL, + long_block_range_error_codes: config.long_block_range_error_codes.clone(), + expected_chain_id: l1_config.identity.chain_id, + }, + l1_config.identity.input_box_address, + l1_config.identity.app_deployment_block, + db_path.clone(), + l1_config.identity.batch_submitter_address, + timing, + // Bootstrap syncs use nested blocking SQLite jobs. The reader takes + // its retained lock clone at construction—not only after worker + // admission—so cancellation of this async command cannot release + // exclusivity beneath an orphaned DB write. + process_lock.clone(), + ); + + tracing::info!( + http_addr = %config.http_addr, + data_dir = %config.data_dir, + eth_rpc_url = %l1_config.eth_rpc_url, + input_box_address = %l1_config.identity.input_box_address, + app_deployment_block = l1_config.identity.app_deployment_block, + chain_id = l1_config.identity.chain_id, + app_address = %l1_config.identity.app_address, + batch_submitter_address = %l1_config.identity.batch_submitter_address, + max_wait_blocks = timing.max_wait_blocks, + preemptive_margin_blocks = timing.preemptive_margin_blocks, + danger_threshold = timing.danger_threshold(), + "sequencer startup" + ); + + // ── Recovery reducer ───────────────────────────────────── + // Local terminal facts are inspected before the first provider call. A + // completed phase always returns through the same reducer before another + // phase or admission. + crate::recovery::run_startup_recovery(&db_path, &mut input_reader, &l1_config, &timing).await?; + + // Setup persisted a real first price, so run performs no synchronous + // fee-source I/O and fee availability never precedes the reducer's local + // terminal-fact inspection. The exhaustive identity match is the worker + // launch decision: fixed mode has no task; Uniswap's supervised worker + // performs the first runtime quote after admission. + let fee_oracle = match l1_config.identity.fee_oracle { + storage::FeeOracleIdentity::Fixed { .. } => None, + storage::FeeOracleIdentity::Uniswap { + weth, + fee_token, + pool, + twap_window_secs, + } => { + let uniswap = crate::l1::fee_oracle::UniswapConfig { + chain_id: l1_config.identity.chain_id, + weth, + fee_token, + pool, + twap_window_secs, + }; + let provider = crate::l1::provider::create_provider( + &config.eth_rpc_url, + config.allow_insecure_rpc, + ) + .map_err(crate::l1::fee_oracle::worker::FeeOracleError::Misconfig)?; + let token = crate::l1::fee_oracle::UniswapV3PriceSource::from_setup_validated( + provider.clone(), + uniswap, + ); + Some(crate::l1::fee_oracle::FeeOracle::new( + db_path.clone(), + Duration::from_millis(config.fee_oracle.poll_interval_ms), + provider, + Box::new(token), + process_lock.clone(), + )) + } + }; + + // ── Prepare → admit → launch ───────────────────────────── + let prepared = PreparedRuntime::::prepare(WorkersConfig { + run_config: config, + l1_config, + timing, + input_reader, + fee_oracle, + process_lock, + }) + .await?; + + // Preparation may take long enough for a clock/view refusal to arise. + // Reinvoke the same reducer over one consistent fact set; workers are + // never launched from an aged decision. + let admission = crate::recovery::admit_runtime(&db_path, &timing)?; + let mut workers = prepared.launch(admission); + + let first_exit = workers.select_first_exit().await; + workers.finish(first_exit).await +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::storage::test_helpers::{record_canonical_divergence, temp_db}; + + #[test] + fn clean_exit_over_persisted_divergence_is_terminal() { + // The one exit code that breaks the supervisor's rediscovery chain + // is 0: divergence recorded during the run must survive a clean + // drain as a terminal verdict, not a normal shutdown. + let db = temp_db("run-clean-exit-divergence"); + let mut storage = + storage::Storage::initialize_for_command(&db.path, LifecycleCommand::Setup) + .expect("initialize"); + record_canonical_divergence(&mut storage, 7, 0); + drop(storage); + + let error = refuse_divergence_on_clean_exit(&db.path) + .expect_err("a clean drain over divergence must not exit 0"); + assert_eq!( + error.exit_code(), + crate::commands::error::EXIT_TERMINAL, + "divergence on the clean path pages" + ); + + let clean = temp_db("run-clean-exit-clean"); + storage::Storage::initialize_for_command(&clean.path, LifecycleCommand::Setup) + .expect("initialize clean"); + refuse_divergence_on_clean_exit(&clean.path).expect("no divergence passes clean"); + } +} diff --git a/sequencer/src/commands/run/startup_hygiene.rs b/sequencer/src/commands/run/startup_hygiene.rs new file mode 100644 index 00000000..0d5256ad --- /dev/null +++ b/sequencer/src/commands/run/startup_hygiene.rs @@ -0,0 +1,259 @@ +// (c) Cartesi and individual authors (see AUTHORS) +// SPDX-License-Identifier: Apache-2.0 (see LICENSE) + +//! Startup snapshot hygiene: the authority-neutral repair pass `prepare` +//! runs over the data directory before the runtime boundary. Synchronous, +//! spawns nothing, awaits nothing, holds no `RuntimeScope` — it runs inside +//! `prepare`, before admission, while zero workers exist. +//! +//! [`run_snapshot_hygiene`] runs five steps, in this order: +//! +//! 1. Reset stale leases. A crashed previous run may have left +//! `lease_count > 0` on dumps that aren't being read by anyone now; +//! without this, GC would skip them forever. +//! 2. Require the finalized snapshot (always-load invariant). `setup` +//! registered the genesis snapshot and `run` gated on atomic setup +//! completion, so it must be present — a missing one is a terminal +//! incomplete-setup, not a cold-start to paper over (run holds no +//! genesis app instance). +//! 3. Re-stamp the finalized dump's `info.toml` from the authoritative DB +//! row. Idempotent, and independent of the two cleanup steps below (the +//! finalized dump is referenced, so neither GC nor the sweep can touch +//! it) — what matters is that it lands before the lane loads the dump, +//! because `info.toml` is the sole authority for `setup --recovery`. +//! Missing or corrupt metadata under a DB-referenced row is terminal, +//! never healed. +//! 4. GC SQLite-side: drop any rows now unreferenced after promotions or +//! invalidations that finalized just before the previous shutdown. +//! 5. Orphan FS sweep: remove directories under `dumps_dir` that aren't +//! tracked by SQLite (crash-during-create_dump or +//! crash-during-GC-after-row-delete artifacts). + +use crate::commands::error::CommandError; +use crate::ingress::inclusion_lane::dump_info::{self, delete_dump_dir}; +use sequencer_core::application::Application; + +/// Run the five-step repair pass (see the module doc for the steps and +/// their ordering). +pub(super) fn run_snapshot_hygiene( + storage: &mut crate::storage::Storage, + dumps_dir: &std::path::Path, +) -> Result<(), CommandError> { + storage.reset_dump_leases()?; + require_finalized_snapshot(storage)?; + restamp_finalized_promotion(storage)?; + let gc_removed = snapshot_gc_at_startup::(storage)?; + let sweep_removed = sweep_orphan_dumps::(storage, dumps_dir)?; + tracing::debug!( + gc_removed, + sweep_removed, + "snapshot startup cleanup complete", + ); + Ok(()) +} + +/// Require the finalized snapshot the lane will `from_dump` against. `setup` +/// registers the genesis snapshot and `run` gates on atomic setup completion, +/// so by the time the lane starts the snapshot must exist. A missing +/// one means the DB's setup is incomplete/corrupt — terminal +/// `SetupNotComplete` (re-run `setup`), not a cold-start to silently heal. +fn require_finalized_snapshot(storage: &mut crate::storage::Storage) -> Result<(), CommandError> { + if storage.finalized_dump()?.is_none() { + return Err(CommandError::Bootstrap( + crate::commands::error::BootstrapError::SetupNotComplete, + )); + } + Ok(()) +} + +/// Re-stamp `B` into the finalized dump's `info.toml` from the +/// authoritative DB row. Idempotent; closes the crash window between a +/// promotion's commit and the lane's in-place stamp. +fn restamp_finalized_promotion(storage: &mut crate::storage::Storage) -> Result<(), CommandError> { + if let Some(finalized) = storage.finalized_dump()? { + let path = finalized.dump.prefix; + dump_info::stamp_promoted_inclusion_block(&path, finalized.inclusion_block) + .map_err(|source| CommandError::ReferencedSnapshotArtifact { path, source })?; + } + Ok(()) +} + +/// Drop any dump rows that are now unreferenced (no pending, no +/// finalized, no leases). The companion `sweep_orphan_dumps` then +/// catches anything on disk that this leaves behind, plus +/// crash-during-create_dump orphans the SQLite layer never saw. +fn snapshot_gc_at_startup( + storage: &mut crate::storage::Storage, +) -> Result { + let removed = storage.gc_unreferenced_dumps()?; + for row in &removed { + if let Err(err) = delete_dump_dir::(&row.prefix) { + tracing::warn!( + error = %err, + prefix = ?row.prefix, + "startup GC: filesystem delete failed; orphan left for sweep", + ); + } + } + Ok(removed.len()) +} + +/// Walk `dumps_dir` and delete any dump directory that isn't in +/// `Storage::list_dump_rows`. Catches: +/// +/// - **crash-during-create**: a dump dir exists on disk (possibly +/// without its app subtree or `info.toml`) but no SQLite row was +/// ever written for it. +/// - **crash-during-GC**: SQLite row was deleted but the filesystem +/// delete either wasn't reached or failed. +/// +/// Filesystem-only — no SQLite writes here. Failures log and +/// continue (the next startup retries). The post-`require_finalized_snapshot` +/// ordering matters: the genesis dump's dir is in +/// `list_dump_rows` by the time this runs, so we never delete it. +fn sweep_orphan_dumps( + storage: &mut crate::storage::Storage, + dumps_dir: &std::path::Path, +) -> Result { + let known: std::collections::HashSet = storage + .list_dump_rows()? + .into_iter() + .map(|row| row.prefix) + .collect(); + let mut removed = 0; + for entry in std::fs::read_dir(dumps_dir)? { + let entry = entry?; + let path = entry.path(); + if known.contains(&path) { + continue; + } + match delete_dump_dir::(&path) { + Ok(()) => removed += 1, + Err(err) => { + tracing::warn!( + error = %err, + ?path, + "orphan dump sweep: delete failed; will retry next startup", + ); + } + } + } + Ok(removed) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::commands::test_support::{SweepTestApp, create_structured_dump}; + use crate::storage::Storage; + use crate::storage::test_helpers::temp_db; + + #[test] + fn startup_restamp_rejects_missing_referenced_snapshot_as_terminal() { + let db = temp_db("restamp-missing-snapshot"); + let mut storage = Storage::open(db.path.as_str()).expect("open"); + let root = tempfile::tempdir().expect("snapshot parent"); + let missing = root.path().join("missing"); + storage + .insert_finalized_dump(&missing, 7, 0) + .expect("register missing fixture"); + + let err = restamp_finalized_promotion(&mut storage) + .expect_err("a durable DB reference cannot point at a missing artifact"); + + assert!(matches!( + &err, + CommandError::ReferencedSnapshotArtifact { .. } + )); + assert_eq!(err.exit_code(), crate::commands::error::EXIT_TERMINAL); + } + + #[test] + fn startup_restamp_rejects_corrupt_referenced_snapshot_as_terminal() { + let db = temp_db("restamp-corrupt-snapshot"); + let mut storage = Storage::open(db.path.as_str()).expect("open"); + let root = tempfile::tempdir().expect("snapshot parent"); + let corrupt = root.path().join("corrupt"); + std::fs::create_dir(&corrupt).expect("create snapshot directory"); + std::fs::write(corrupt.join("info.toml"), "not = valid = toml") + .expect("write corrupt metadata"); + storage + .insert_finalized_dump(&corrupt, 7, 0) + .expect("register corrupt fixture"); + + let err = restamp_finalized_promotion(&mut storage) + .expect_err("corrupt durable metadata cannot be retried as operational I/O"); + + assert!(matches!( + &err, + CommandError::ReferencedSnapshotArtifact { .. } + )); + assert_eq!(err.exit_code(), crate::commands::error::EXIT_TERMINAL); + } + + #[test] + fn sweep_orphan_dumps_removes_directories_not_in_storage() { + let db = temp_db("sweep-orphans"); + let mut storage = Storage::open(db.path.as_str()).expect("open"); + let dumps_dir = tempfile::tempdir().expect("dumps dir"); + + // Tracked dump (in SQLite). + let tracked = dumps_dir.path().join("tracked"); + create_structured_dump(&tracked); + storage + .insert_finalized_dump(&tracked, 0, 0) + .expect("register tracked"); + + // Two orphans (NOT in SQLite). One is fully formed; the other + // mimics a crash between dir creation and the app dump (no + // `state` subtree) — the sweep must remove both. + let orphan_a = dumps_dir.path().join("orphan-a"); + let orphan_b = dumps_dir.path().join("orphan-b"); + create_structured_dump(&orphan_a); + std::fs::create_dir(&orphan_b).expect("orphan b dir"); + + let removed = sweep_orphan_dumps::(&mut storage, dumps_dir.path()).unwrap(); + assert_eq!(removed, 2); + assert!(tracked.exists(), "tracked dump must survive"); + assert!(!orphan_a.exists()); + assert!(!orphan_b.exists()); + } + + #[test] + fn sweep_orphan_dumps_on_empty_directory_is_noop() { + let db = temp_db("sweep-empty"); + let mut storage = Storage::open(db.path.as_str()).expect("open"); + let dumps_dir = tempfile::tempdir().expect("dumps dir"); + + let removed = sweep_orphan_dumps::(&mut storage, dumps_dir.path()).unwrap(); + assert_eq!(removed, 0); + } + + #[test] + fn snapshot_gc_at_startup_removes_unreferenced_rows() { + let db = temp_db("gc-startup"); + let mut storage = Storage::open(db.path.as_str()).expect("open"); + let dumps_dir = tempfile::tempdir().expect("dumps dir"); + + // Two dumps: superseded + finalized. + let superseded = dumps_dir.path().join("superseded"); + let finalized = dumps_dir.path().join("finalized"); + create_structured_dump(&superseded); + create_structured_dump(&finalized); + storage + .insert_pending_dump(&superseded, 0, 0) + .expect("pending 0"); + storage.promote_finalized(0, 0).expect("promote 0"); + storage + .insert_pending_dump(&finalized, 1, 0) + .expect("pending 1"); + storage.promote_finalized(1, 0).expect("promote 1"); + // `superseded`'s row is now unreferenced (replaced by + // finalized's promotion), but the directory is still on disk. + + let removed = snapshot_gc_at_startup::(&mut storage).unwrap(); + assert_eq!(removed, 1); + assert!(!superseded.exists(), "GC removed the superseded directory"); + assert!(finalized.exists(), "current finalized survived"); + } +} diff --git a/sequencer/src/commands/run/workers.rs b/sequencer/src/commands/run/workers.rs new file mode 100644 index 00000000..3f0b6979 --- /dev/null +++ b/sequencer/src/commands/run/workers.rs @@ -0,0 +1,1425 @@ +// (c) Cartesi and individual authors (see AUTHORS) +// SPDX-License-Identifier: Apache-2.0 (see LICENSE) + +//! Runtime worker lifecycle: prepare → admit → launch → orderly cleanup. +//! +//! [`Workers`] owns the core runtime worker handles plus an optional live +//! Uniswap fee-oracle worker; fixed pricing has no worker. +//! Runtime construction has one explicit authority boundary: +//! +//! - [`PreparedRuntime::prepare`]: prepare every fallible or awaited +//! dependency while launching zero workers. +//! - [`PreparedRuntime::launch`]: consume the controller's single-use durable +//! admission witness and launch all workers in one infallible, +//! non-yielding step, returning the owning struct. +//! - [`Workers::select_first_exit`]: race the workers + OS shutdown signal, +//! return whichever fired first. +//! - [`Workers::finish`]: request shutdown, race all remaining components to +//! completion (so a hung drain cannot hide a terminal exit), and surface the +//! primary failure. +//! +//! Worker plumbing is intentionally explicit per-worker (6 fields, 6 spawn +//! statements, 6 select arms, 6 cleanup entries). Adding a seventh worker +//! means editing each of those four sites, and each is compile-forced: the +//! `Workers` literal in `launch`, and the exhaustive `let Self { .. }` +//! destructures in `select_first_exit` and `finish` — where a bound-but- +//! unused field fails CI under `-D warnings`. Keep those destructures +//! exhaustive (no `..`): they are the enforcement, not style. + +use std::future::Future; +use std::marker::PhantomData; +use std::pin::Pin; +use std::sync::Arc; +use std::task::Poll; +use std::time::Duration; + +use alloy::providers::DynProvider; +use tokio::task::JoinHandle; +use tracing::warn; + +use crate::commands::config::RunConfig; +use crate::commands::error::{CommandError, WorkerExit, WorkerStop}; +use crate::egress::l2_tx_feed::{L2TxFeed, L2TxFeedConfig}; +use crate::http::{self, ApiConfig}; +use crate::ingress::inclusion_lane::{ + InclusionLane, InclusionLaneConfig, InclusionLaneError, dump_info, +}; +use crate::l1::L1Config; +use crate::l1::fee_oracle::FeeOracle; +use crate::l1::reader::{InputReader, InputReaderError}; +use crate::l1::submitter::{ + BatchPosterConfig, BatchSubmitter, BatchSubmitterConfig, BatchSubmitterError, + EthereumBatchPoster, SubmitterExit, +}; +use crate::recovery::{DangerDetector, DangerDetectorError, DetectorExit}; +use crate::runtime::process_lock::ProcessLock; +use crate::runtime::shutdown::RuntimeScope; +use sequencer_core::application::Application; +use sequencer_core::protocol::ProtocolTiming; + +const QUEUE_CAPACITY: usize = 8192; +/// Danger detector cadence. Cheap DB-only check; re-running quickly bounds the +/// lag on entering the danger zone. The preemptive margin absorbs bounded lag. +const DANGER_DETECTOR_POLL_INTERVAL: Duration = Duration::from_secs(2); + +/// Which event ended the `select!` race in [`Workers::select_first_exit`]. +pub(super) enum FirstExit { + Signal(Option), + Worker(WorkerExit), + /// A terminal fault was contained by a runtime component. The black-box + /// terminal-cause row was attempted but is best-effort telemetry; the + /// exit code and logs carry the verdict if it failed. + Contained, +} + +/// Inputs to [`PreparedRuntime::prepare`]. Consumed entirely; the caller has +/// nothing further to do with these after the call. +/// +/// Everything here is built by `run` because the recovery reducer consumes +/// it or must run after it; anything the workers alone need is derived +/// inside `prepare`. +/// +/// No genesis app instance: `setup` already registered the finalized genesis +/// snapshot, so the lane reloads via `A::from_dump`. No EIP-712 domain +/// either: `prepare` derives it from the pinned deployment identity that +/// `l1_config` carries verbatim. +pub(super) struct WorkersConfig { + pub run_config: RunConfig, + pub l1_config: L1Config, + pub timing: ProtocolTiming, + pub input_reader: InputReader, + /// Launch-ready without source I/O: setup supplied the persisted price, + /// and a Uniswap worker quotes on its first supervised iteration. Fixed + /// pricing has no worker (`None`). + pub fee_oracle: Option, + /// Exclusive data-directory ownership, acquired before bootstrap and + /// transferred into the runtime lifetime at worker admission. + pub process_lock: ProcessLock, +} + +/// Requests shutdown if construction or runtime ownership is dropped; a panic +/// instead enters terminal containment before unwind can strand runtime work. +/// Every spawned worker retains a [`RuntimeScope`] clone, which also retains +/// the process lock, so exclusivity remains held until the workers finish even +/// though Drop cannot join asynchronously. +struct ShutdownOnDrop(RuntimeScope); + +impl Drop for ShutdownOnDrop { + fn drop(&mut self) { + if std::thread::panicking() { + // A panic in the admitted runtime controller is a trusted-code + // failure just like a worker panic. Contain while the runtime + // lifetime is still owned so the terminal watchdog bounds any + // worker or detached blocking operation left behind by unwind. + self.0 + .contain_storage_invariant_failure("runtime controller panicked"); + } else { + self.0.request_shutdown(); + } + } +} + +/// Fully prepared runtime state: exactly the arguments `launch` hands to +/// the workers. Configuration is consumed by `prepare`; only launch-ready +/// values cross the authority boundary. Owning this value launches no tasks +/// and grants no sequencing authority. +pub(super) struct PreparedRuntime { + input_reader: InputReader, + api_config: ApiConfig, + fee_oracle: Option, + storage: crate::storage::Storage, + lane_config: InclusionLaneConfig, + submitter: BatchSubmitter, + detector: DangerDetector, + tx_feed: L2TxFeed, + listener: tokio::net::TcpListener, + bound_addr: std::net::SocketAddr, + snapshot_state: http::SnapshotState, + shutdown: RuntimeScope, + shutdown_on_drop: ShutdownOnDrop, + /// No `A` *value* is ever held — `setup` registered the genesis snapshot + /// and the lane reloads via `A::from_dump`. The parameter feeds the + /// lane's `start`, the payload bound, and the snapshot path hook. + _application: PhantomData A>, +} + +/// Owns the runtime worker handles + the shutdown signal that drives them. +/// [`PreparedRuntime::launch`] and teardown ([`Workers::finish`]) bracket the +/// worker lifecycle. +pub(super) struct Workers { + server: JoinHandle>, + lane: JoinHandle>, + reader: JoinHandle>, + submitter: JoinHandle>, + detector: JoinHandle>, + fee_oracle: Option>>, + shutdown: RuntimeScope, + _shutdown_on_drop: ShutdownOnDrop, +} + +impl PreparedRuntime { + /// Prepare every fallible or awaited runtime dependency while launching + /// zero tasks. Durable admission remains the controller's responsibility. + pub(super) async fn prepare(cfg: WorkersConfig) -> Result { + let WorkersConfig { + run_config, + l1_config, + timing, + input_reader, + fee_oracle, + process_lock, + } = cfg; + + // Derived values — kept inside `prepare` so `WorkersConfig` stays + // minimal and these aren't computed twice in the caller. + let db_path = run_config.db_path(); + // The EIP-712 domain is the signature-verification boundary, so it + // is derived from exactly one source: the pinned deployment identity + // that `l1_config` carries verbatim out of `load_setup_identity`. + let domain = sequencer_core::build_input_domain( + l1_config.identity.chain_id, + l1_config.identity.app_address, + ); + + // The scope is the runtime-lifetime capability: every worker + // receives a clone, and every clone keeps the process lock alive. + // The drop guard requests shutdown on any partial-construction `?`, + // panic unwind, or cancellation of the owning `run` future. + let shutdown = RuntimeScope::new(process_lock); + let shutdown_on_drop = ShutdownOnDrop(shutdown.clone()); + // Durable terminal-fault recorder: best-effort telemetry. A + // successful write appends the black-box terminal-cause row; a + // failed write loses only the black-box copy — the exit code and + // logs still carry the verdict, and a persistent fault re-detects + // fail-loud on the next boot that reads it. + install_terminal_fault_recorder(&shutdown, db_path.clone()); + + let mut storage = crate::storage::Storage::open(&db_path)?; + let dumps_dir = std::path::Path::new(&run_config.data_dir).join("dumps"); + std::fs::create_dir_all(&dumps_dir)?; + + // Authority-neutral snapshot repair before the boundary; the five + // order-critical steps are documented in `startup_hygiene`. + super::startup_hygiene::run_snapshot_hygiene::(&mut storage, &dumps_dir)?; + + // Prepare every remaining fallible or awaited dependency before the + // authority boundary. Cancellation observes zero workers. + input_reader.preflight_storage()?; + + let poster_config = BatchPosterConfig { + l1_submit_address: l1_config.identity.input_box_address, + app_address: l1_config.identity.app_address, + batch_submitter_address: l1_config.identity.batch_submitter_address, + start_block: l1_config.identity.app_deployment_block, + confirmation_depth: run_config.batch_submitter_confirmation_depth, + seconds_per_block: timing.seconds_per_block, + long_block_range_error_codes: run_config.long_block_range_error_codes.clone(), + expected_chain_id: l1_config.identity.chain_id, + }; + let provider = build_batch_submitter_provider(&l1_config)?; + let poster = Arc::new(EthereumBatchPoster::new( + provider, + poster_config, + shutdown.clone(), + )); + let submitter_config = BatchSubmitterConfig { + idle_poll_interval_ms: run_config.batch_submitter_idle_poll_interval_ms, + }; + let submitter = BatchSubmitter::new( + db_path.clone(), + poster, + submitter_config, + shutdown.process_lock(), + ); + submitter.preflight_storage()?; + + let detector = DangerDetector::new( + db_path.clone(), + timing, + DANGER_DETECTOR_POLL_INTERVAL, + shutdown.process_lock(), + ); + detector.preflight_storage()?; + + let tx_feed = L2TxFeed::new( + db_path.clone(), + shutdown.clone(), + L2TxFeedConfig::new(l1_config.identity.batch_submitter_address), + ); + + // Configuration ends here: the remaining values are exactly what + // `launch` hands to the workers, so the config structs never cross + // the authority boundary. + let lane_config = + InclusionLaneConfig::new(l1_config.identity.batch_submitter_address, dumps_dir) + .with_max_batch_open(run_config.max_batch_open()); + let api_config = ApiConfig::new(domain, A::MAX_METHOD_PAYLOAD_BYTES); + let listener = tokio::net::TcpListener::bind(&run_config.http_addr).await?; + let bound_addr = listener.local_addr()?; + let snapshot_state = http::SnapshotState { + db_path, + // The DB row stores the dump *directory*; the app's state + // file lives under its `state` subtree. + state_file_in_dump: |dump_dir| A::state_file_in_dump(&dump_info::app_prefix(dump_dir)), + }; + + Ok(Self { + input_reader, + api_config, + fee_oracle, + storage, + lane_config, + submitter, + detector, + tx_feed, + listener, + bound_addr, + snapshot_state, + shutdown, + shutdown_on_drop, + _application: PhantomData, + }) + } + + /// Launch all workers in one infallible, non-async, non-yielding step. + /// Consuming the [`crate::recovery::RuntimeAdmission`] witness here is + /// what gates launching on the reducer's fresh clean decision — the + /// witness's sole constructor is `admit_runtime`, and launch uses it up. + pub(super) fn launch(self, _admission: crate::recovery::RuntimeAdmission) -> Workers { + let Self { + input_reader, + api_config, + fee_oracle, + storage, + lane_config, + submitter, + detector, + tx_feed, + listener, + bound_addr, + snapshot_state, + shutdown, + shutdown_on_drop, + _application: _, + } = self; + + let (tx, lane) = + InclusionLane::::start(QUEUE_CAPACITY, shutdown.clone(), storage, lane_config); + let reader = input_reader.start_preflighted(shutdown.clone()); + let submitter = submitter.start_preflighted(shutdown.clone()); + let detector = detector.start_preflighted(shutdown.clone()); + let fee_oracle = fee_oracle.map(|oracle| oracle.start(shutdown.clone())); + // HTTP server (ingress /tx + egress /ws/subscribe + /health, currently merged). + let server = http::start_on_listener( + listener, + tx, + shutdown.clone(), + tx_feed, + api_config, + snapshot_state, + ); + tracing::info!(address = %bound_addr, "listening"); + + Workers { + server, + lane, + reader, + submitter, + detector, + fee_oracle, + shutdown, + _shutdown_on_drop: shutdown_on_drop, + } + } +} + +impl Workers { + /// Race an OS shutdown signal against each worker's join handle. The first to complete + /// produces the [`FirstExit`]. + pub(super) async fn select_first_exit(&mut self) -> FirstExit { + // Exhaustive destructure (no `..`): a new worker field fails to + // compile here, so it cannot be forgotten in the race below — the + // same forcing `finish`'s destructure provides for cleanup. + let Self { + server, + lane, + reader, + submitter, + detector, + fee_oracle, + shutdown, + _shutdown_on_drop: _, + } = self; + let shutdown_signal = os_shutdown_signal(); + tokio::pin!(shutdown_signal); + tokio::select! { + biased; + _ = shutdown.wait_for_shutdown() => { + if shutdown.is_storage_invariant_contained() { + FirstExit::Contained + } else { + // Externally requested shutdown without a contained + // fault: treated like a signal-driven drain. + FirstExit::Signal(None) + } + } + signal_result = &mut shutdown_signal => FirstExit::signal(signal_result), + server_result = &mut *server => + FirstExit::Worker(WorkerExit::Server(WorkerStop::from_select(server_result))), + lane_result = &mut *lane => + FirstExit::Worker(WorkerExit::Lane(WorkerStop::from_select(lane_result))), + reader_result = &mut *reader => + FirstExit::Worker(WorkerExit::InputReader(WorkerStop::from_select(reader_result))), + submitter_result = &mut *submitter => + FirstExit::Worker(WorkerExit::BatchSubmitter(WorkerStop::from_select( + // A worker returning `Shutdown` outside a real shutdown + // means it stopped on its own — the unexpected case. + submitter_result.map(|r| r.map(|SubmitterExit::Shutdown| ())), + ))), + detector_result = &mut *detector => FirstExit::detector(detector_result), + fee_oracle_result = async { + match fee_oracle.as_mut() { + Some(handle) => handle.await, + // Fixed mode: no oracle worker, so this arm never resolves. + None => std::future::pending().await, + } + } => FirstExit::Worker(WorkerExit::FeeOracle(WorkerStop::from_select( + fee_oracle_result, + ))), + } + } + + /// Drive orderly cleanup: request shutdown, poll all workers concurrently + /// to completion, and surface the primary failure. A sticky storage + /// invariant fault or terminal cleanup error always takes precedence over + /// an earlier nonterminal worker/signal result. Concurrent polling matters: + /// a hung drain must not hide a terminal exit that arms the hard watchdog. + pub(super) async fn finish(self, first_exit: FirstExit) -> Result<(), CommandError> { + match &first_exit { + // Already contained by the raising component; its best-effort + // terminal-cause journal append was attempted there. + FirstExit::Contained => {} + FirstExit::Worker(exit) if exit.is_terminal() => { + // Log the typed exit here: the terminal return path below + // reports the invariant-violation class, and the cause must + // not be flattened out of the operator's view. + tracing::error!( + component = exit.worker_id().label(), + error = %exit, + "terminal worker exit; containing runtime" + ); + self.shutdown.contain_storage_invariant_failure(format!( + "terminal {} worker exit: {exit}", + exit.worker_id().label() + )); + } + FirstExit::Signal(_) | FirstExit::Worker(_) => { + self.shutdown.request_shutdown(); + } + } + + let Self { + server, + lane, + reader, + submitter, + detector, + fee_oracle, + shutdown, + _shutdown_on_drop, + } = self; + let mut components: Vec<(WorkerId, ComponentShutdown)> = vec![ + (WorkerId::Server, Box::pin(wait_for_server_shutdown(server))), + (WorkerId::Lane, Box::pin(wait_for_lane_shutdown(lane))), + ( + WorkerId::InputReader, + Box::pin(wait_for_input_reader_shutdown(reader)), + ), + ( + WorkerId::BatchSubmitter, + Box::pin(wait_for_batch_submitter_shutdown(submitter)), + ), + ( + WorkerId::DangerDetector, + Box::pin(wait_for_danger_detector_shutdown(detector)), + ), + ]; + if let Some(fee_oracle) = fee_oracle { + components.push(( + WorkerId::FeeOracle, + Box::pin(wait_for_fee_oracle_shutdown(fee_oracle)), + )); + } + + // One drain, two phases: + // - "cleanup-time" (worker failure): the primary is already in hand; + // every OTHER component is awaited for orderly cleanup. + // - "shutdown-time" (signal or already-contained): everything drains; + // the signal handler's own error outranks any later component error. + let (worker_failure, signal_error): (Option<(WorkerId, WorkerExit)>, Option) = + match first_exit { + FirstExit::Signal(err) => (None, err), + FirstExit::Worker(exit) => { + let id = exit.worker_id(); + (Some((id, exit)), None) + } + FirstExit::Contained => (None, None), + }; + + if let Some((failed, _)) = &worker_failure { + let failed_index = components + .iter() + .position(|(id, _)| id == failed) + .expect("primary worker must be present in the cleanup set"); + // Drop the primary's future without awaiting — its task is + // already done (it tripped the select) and re-polling a completed + // JoinHandle panics. Its typed exit is surfaced by the precedence + // match below. + drop(components.swap_remove(failed_index)); + } + + let phase = if worker_failure.is_some() { + "cleanup-time" + } else { + "shutdown-time" + }; + + // Await EVERY remaining component — in both phases — so each worker's + // JoinHandle is joined and its task fully drains. A `break` here would + // drop the remaining components' futures un-awaited, which DETACHES + // those tasks (only `JoinHandle::abort()` cancels a dropped handle) — + // they'd be killed mid-drain at runtime teardown, the exact + // abrupt-write case the startup snapshot hygiene (sweep/gc/re-stamp) + // exists to clean up after. The primary removed above is the one + // deliberate exception: its task already completed, so awaiting it + // would panic rather than drain anything. Keep the first ordinary + // error to surface; terminal errors contain instead. + let mut drain_error: Option = None; + while let Some((id, result)) = next_component_shutdown(&mut components).await { + if let Err(e) = result { + warn!( + component = id.label(), + phase, + error = %e, + "component errored during runtime drain" + ); + if e.is_terminal() { + shutdown.contain_storage_invariant_failure(format!( + "{phase} {} worker exit: {e}", + id.label() + )); + } else if drain_error.is_none() { + drain_error = Some(e); + } + } + } + + // The single precedence site: contained > primary > signal > first + // drain error. + match ( + contained_verdict(&shutdown), + worker_failure, + signal_error, + drain_error, + ) { + (Some(contained), ..) => Err(contained), + (None, Some((_, primary_exit)), _, _) => Err(CommandError::Worker(primary_exit)), + (None, None, Some(signal_err), _) => Err(signal_err), + (None, None, None, Some(exit)) => Err(CommandError::Worker(exit)), + (None, None, None, None) => Ok(()), + } + } +} + +fn install_terminal_fault_recorder(shutdown: &RuntimeScope, db_path: String) { + shutdown.set_fault_recorder(Arc::new(move |cause: &str| { + match crate::storage::Storage::open_writer(&db_path) { + Ok(mut storage) => { + if let Err(err) = + storage.record_terminal_fault(crate::storage::LifecycleCommand::Run, cause) + { + tracing::warn!(error = %err, "terminal cause not recorded in the black box; the exit code and this log carry the verdict"); + } + } + Err(err) => { + tracing::warn!(error = %err, "terminal cause not recorded in the black box (storage open failed); the exit code and this log carry the verdict"); + } + } + })); +} + +async fn os_shutdown_signal() -> std::io::Result<()> { + #[cfg(unix)] + { + let mut terminate = + tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())?; + tokio::select! { + result = tokio::signal::ctrl_c() => result, + _ = terminate.recv() => Ok(()), + } + } + #[cfg(not(unix))] + { + tokio::signal::ctrl_c().await + } +} + +/// Stable identity of each long-lived worker. The `finish` worker-failure path +/// skips the already-exited worker by matching on this enum, not on a label +/// string that could silently drift from the component-array order. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum WorkerId { + Server, + Lane, + InputReader, + BatchSubmitter, + DangerDetector, + FeeOracle, +} + +impl WorkerId { + /// Human-readable label for logs, matching the `Workers::finish` list. + fn label(self) -> &'static str { + match self { + WorkerId::Server => "server", + WorkerId::Lane => "inclusion lane", + WorkerId::InputReader => "input reader", + WorkerId::BatchSubmitter => "batch submitter", + WorkerId::DangerDetector => "danger detector", + WorkerId::FeeOracle => "fee oracle", + } + } +} + +impl WorkerExit { + /// Which worker produced this exit. + fn worker_id(&self) -> WorkerId { + match self { + WorkerExit::Server(_) => WorkerId::Server, + WorkerExit::Lane(_) => WorkerId::Lane, + WorkerExit::InputReader(_) => WorkerId::InputReader, + WorkerExit::BatchSubmitter(_) => WorkerId::BatchSubmitter, + WorkerExit::DangerDetector(_) | WorkerExit::DangerDetected { .. } => { + WorkerId::DangerDetector + } + WorkerExit::FeeOracle(_) => WorkerId::FeeOracle, + } + } +} + +// ── FirstExit constructors ──────────────────────────────────────────── +// +// Named constructors, not `From` impls keyed on the worker's result type: +// two workers sharing a result type would silently misroute through blanket +// dispatch, and the select arms should say which worker they map. + +impl FirstExit { + /// ctrl_c shutdown signal: `Ok(())` = clean signal, `Err(io)` = + /// signal-handler installation failed. + fn signal(result: Result<(), std::io::Error>) -> Self { + FirstExit::Signal(result.err().map(CommandError::from)) + } + + /// The detector's `RecoveryRequired` trip is a first-class exit, not an + /// error; `Shutdown` outside a real shutdown means it stopped on its own. + fn detector( + result: Result, tokio::task::JoinError>, + ) -> Self { + let stop = match result { + Ok(Ok(DetectorExit::RecoveryRequired { status })) => { + return FirstExit::Worker(WorkerExit::DangerDetected { status }); + } + Ok(Ok(DetectorExit::Shutdown)) => WorkerStop::StoppedUnexpectedly, + Ok(Err(source)) => WorkerStop::Source(source), + Err(source) => WorkerStop::Join(source), + }; + FirstExit::Worker(WorkerExit::DangerDetector(stop)) + } +} + +// ── Shutdown waiters ─────────────────────────────────────────────────── +// +// Each component future awaits a worker's JoinHandle and converts via +// `WorkerStop::from_shutdown` (which knows `Ok` is the graceful drain). + +type ComponentShutdown = Pin> + Send>>; + +/// Observe whichever remaining component finishes next. Cleanup must poll all +/// workers concurrently: otherwise one hung component can hide a terminal +/// panic/invariant exit from a later slot forever, preventing containment and +/// its hard abort bound from ever arming. +/// +/// No `.await` may separate the inner `Poll::Ready` from the `swap_remove` — +/// a cancellation in that window would leave a completed future in the set, +/// and the next poll of it panics. `swap_remove` also reorders the set, so +/// completion order among concurrently-ready components is unspecified; only +/// terminal exits carry precedence. +async fn next_component_shutdown( + components: &mut Vec<(WorkerId, ComponentShutdown)>, +) -> Option<(WorkerId, Result<(), WorkerExit>)> { + if components.is_empty() { + return None; + } + + let (ready_index, result) = std::future::poll_fn(|cx| { + for (index, (_, component)) in components.iter_mut().enumerate() { + if let Poll::Ready(result) = component.as_mut().poll(cx) { + return Poll::Ready((index, result)); + } + } + Poll::Pending + }) + .await; + let (id, _) = components.swap_remove(ready_index); + Some((id, result)) +} + +/// The single post-cleanup containment check: if a terminal fault was +/// contained anywhere (primary, cleanup, or a non-worker component), surface +/// the terminal class. The cause is present whenever containment reads true +/// (they are one `OnceLock`); recorder failure loses only the journal's +/// telemetry copy of the cause, never the sticky in-process verdict or the +/// terminal exit class. +fn contained_verdict(shutdown: &RuntimeScope) -> Option { + shutdown + .containment_cause() + .map(|cause| CommandError::StorageInvariantViolation { + cause: cause.to_string(), + }) +} + +async fn wait_for_server_shutdown( + server_task: JoinHandle>, +) -> Result<(), WorkerExit> { + WorkerStop::from_shutdown(server_task.await).map_err(WorkerExit::Server) +} + +async fn wait_for_lane_shutdown( + handle: JoinHandle>, +) -> Result<(), WorkerExit> { + WorkerStop::from_shutdown(handle.await).map_err(WorkerExit::Lane) +} + +async fn wait_for_input_reader_shutdown( + handle: JoinHandle>, +) -> Result<(), WorkerExit> { + WorkerStop::from_shutdown(handle.await).map_err(WorkerExit::InputReader) +} + +async fn wait_for_batch_submitter_shutdown( + handle: JoinHandle>, +) -> Result<(), WorkerExit> { + WorkerStop::from_shutdown(handle.await.map(|r| r.map(|SubmitterExit::Shutdown| ()))) + .map_err(WorkerExit::BatchSubmitter) +} + +/// Detector `Shutdown` is the graceful drain; a `RecoveryRequired` trip +/// during drain still surfaces as the first-class danger exit. +async fn wait_for_danger_detector_shutdown( + handle: JoinHandle>, +) -> Result<(), WorkerExit> { + match handle.await { + Ok(Ok(DetectorExit::Shutdown)) => Ok(()), + Ok(Ok(DetectorExit::RecoveryRequired { status })) => { + Err(WorkerExit::DangerDetected { status }) + } + Ok(Err(source)) => Err(WorkerExit::DangerDetector(WorkerStop::Source(source))), + Err(source) => Err(WorkerExit::DangerDetector(WorkerStop::Join(source))), + } +} + +async fn wait_for_fee_oracle_shutdown( + handle: JoinHandle>, +) -> Result<(), WorkerExit> { + WorkerStop::from_shutdown(handle.await).map_err(WorkerExit::FeeOracle) +} + +// Built once during preparation (sync, raw `create_signer_provider`). The +// submitter is long-lived, so a one-shot startup chain-id check would go stale; the +// keyed-write guard instead lives in `EthereumBatchPoster::submit_batches`, +// which re-confirms the chain id immediately before every productive send. +// The key was already verified against the pinned submitter, so a build +// failure here is a bad RPC URL / client misconfiguration — the same +// deterministic terminal class as every signer misconfig. +fn build_batch_submitter_provider(l1: &L1Config) -> Result { + crate::l1::provider::create_signer_provider( + &l1.eth_rpc_url, + l1.batch_submitter_private_key.expose_secret(), + l1.allow_insecure_rpc, + ) + .map_err(|message| { + CommandError::Bootstrap(crate::commands::error::BootstrapError::SignerMisconfig { message }) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::recovery::DangerDetectorError; + use crate::storage::DangerStatus; + use clap::Parser; + + #[derive(Clone, Default)] + struct StartupProbeApp { + progress: sequencer_core::application::ApplicationProgress, + } + + impl Application for StartupProbeApp { + const MAX_METHOD_PAYLOAD_BYTES: usize = 0; + + fn validate_user_op( + &self, + _sender: alloy_primitives::Address, + _user_op: &sequencer_core::user_op::UserOp, + _current_fee: u16, + ) -> Result<(), sequencer_core::application::InvalidReason> { + Ok(()) + } + + fn apply_valid_user_op( + &mut self, + _capability: sequencer_core::application::ApplyInputCapability<'_>, + _user_op: &sequencer_core::l2_tx::ValidUserOp, + _safe_block: u64, + ) -> Result + { + Ok(Vec::new()) + } + + fn apply_direct_input( + &mut self, + _capability: sequencer_core::application::ApplyInputCapability<'_>, + _input: &sequencer_core::l2_tx::DirectInput, + ) -> Result + { + Ok(Vec::new()) + } + + fn execution_progress(&self) -> &sequencer_core::application::ApplicationProgress { + &self.progress + } + + fn execution_progress_mut( + &mut self, + _capability: sequencer_core::application::ProgressCommitCapability<'_>, + ) -> &mut sequencer_core::application::ApplicationProgress { + &mut self.progress + } + + fn from_dump( + _prefix: &std::path::Path, + ) -> Result { + Ok(Self::default()) + } + + fn create_dump( + &self, + prefix: &std::path::Path, + ) -> Result<(), sequencer_core::application::AppError> { + std::fs::create_dir(prefix)?; + std::fs::write(prefix.join("state"), [])?; + Ok(()) + } + + fn delete_dump( + prefix: &std::path::Path, + ) -> Result<(), sequencer_core::application::AppError> { + std::fs::remove_dir_all(prefix)?; + Ok(()) + } + + fn state_file_in_dump(prefix: &std::path::Path) -> std::path::PathBuf { + prefix.join("state") + } + } + + // ── select!-arm `From` conversions ────────────────── + // + // The detector arm is the interesting one (DangerDetected vs Shutdown vs + // Source vs Join). The other workers follow a uniform 3-way mapping + // covered by the type system. + + type DetectorJoinResult = + Result, tokio::task::JoinError>; + + #[test] + fn detector_shutdown_in_select_maps_to_stopped_unexpectedly() { + let result: DetectorJoinResult = Ok(Ok(DetectorExit::Shutdown)); + assert!(matches!( + FirstExit::detector(result), + FirstExit::Worker(WorkerExit::DangerDetector(WorkerStop::StoppedUnexpectedly)) + )); + } + + #[test] + fn detector_recovery_required_maps_to_danger_detected() { + let result: DetectorJoinResult = Ok(Ok(DetectorExit::RecoveryRequired { + status: DangerStatus::ClosedBatchInDanger(7), + })); + assert!(matches!( + FirstExit::detector(result), + FirstExit::Worker(WorkerExit::DangerDetected { + status: DangerStatus::ClosedBatchInDanger(7) + }) + )); + } + + #[test] + fn production_recorder_poison_is_durable_and_first_writer_wins() { + let db = temp_db("runtime-lifecycle-recorder"); + let mut storage = + Storage::initialize_for_command(&db.path, crate::storage::LifecycleCommand::Setup) + .expect("initialize setup"); + storage + .insert_initial_finalized_dump(&db._dir.path().join("finalized"), 0, 0, 0, 0) + .expect("register finalized snapshot"); + storage.complete_setup().expect("complete setup"); + drop(storage); + let shutdown = RuntimeScope::default(); + install_terminal_fault_recorder(&shutdown, db.path.clone()); + + shutdown.contain_storage_invariant_failure("first terminal cause"); + shutdown.contain_storage_invariant_failure("echo"); + + let fault = Storage::open_read_only(&db.path) + .expect("reopen") + .latest_terminal_fault() + .expect("read") + .expect("recorded fault"); + assert_eq!(fault.command, crate::storage::LifecycleCommand::Run); + assert_eq!(fault.cause, "first terminal cause"); + } + + /// Workers whose tasks idle until shutdown. The shared signal retains the + /// first cause independently of the best-effort durable recorder. + fn waiting_workers(shutdown: &RuntimeScope) -> (Workers, tempfile::TempDir) { + let dir = tempfile::tempdir().expect("workers tempdir"); + let server = tokio::spawn({ + let shutdown = shutdown.clone(); + async move { + shutdown.wait_for_shutdown().await; + Ok(()) + } + }); + let lane = tokio::spawn({ + let shutdown = shutdown.clone(); + async move { + shutdown.wait_for_shutdown().await; + Ok(()) + } + }); + let reader = tokio::spawn({ + let shutdown = shutdown.clone(); + async move { + shutdown.wait_for_shutdown().await; + Ok(()) + } + }); + let submitter = tokio::spawn({ + let shutdown = shutdown.clone(); + async move { + shutdown.wait_for_shutdown().await; + Ok(SubmitterExit::Shutdown) + } + }); + let detector = tokio::spawn({ + let shutdown = shutdown.clone(); + async move { + shutdown.wait_for_shutdown().await; + Ok(DetectorExit::Shutdown) + } + }); + ( + Workers { + server, + lane, + reader, + submitter, + detector, + fee_oracle: None, + shutdown: shutdown.clone(), + _shutdown_on_drop: ShutdownOnDrop(shutdown.clone()), + }, + dir, + ) + } + + #[tokio::test] + async fn dropped_runtime_scope_keeps_lock_until_detached_worker_stops() { + let dir = tempfile::tempdir().expect("runtime data dir"); + let data_dir = dir.path().to_str().expect("utf8 path"); + let process_lock = ProcessLock::acquire(data_dir).expect("acquire runtime lock"); + let shutdown = RuntimeScope::new(process_lock); + let shutdown_on_drop = ShutdownOnDrop(shutdown.clone()); + let (stopped_tx, stopped_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = tokio::sync::oneshot::channel(); + + // Model a worker spawned before a later startup step fails. Its join + // handle is dropped (Tokio detaches it), but its signal clone keeps + // process ownership until it has observed shutdown and finished. + let worker_shutdown = shutdown.clone(); + let worker = tokio::spawn(async move { + worker_shutdown.wait_for_shutdown().await; + let _ = release_rx.await; + drop(worker_shutdown); + let _ = stopped_tx.send(()); + }); + drop(worker); + + drop(shutdown_on_drop); + drop(shutdown); + let refused = ProcessLock::acquire(data_dir) + .expect_err("a detached but live worker must retain process ownership"); + assert!(matches!( + refused, + crate::runtime::process_lock::ProcessLockError::Locked { .. } + )); + + release_tx.send(()).expect("release worker"); + stopped_rx.await.expect("worker stopped"); + ProcessLock::acquire(data_dir).expect("lock releases after the last worker exits"); + } + + #[test] + fn controller_panic_is_contained_before_scope_unwind_finishes() { + let shutdown = RuntimeScope::default(); + let guard = ShutdownOnDrop(shutdown.clone()); + + let panic = std::panic::catch_unwind(std::panic::AssertUnwindSafe(move || { + let _guard = guard; + panic!("controller panic probe"); + })); + + assert!(panic.is_err()); + assert!(shutdown.is_storage_invariant_contained()); + assert!(shutdown.is_shutdown_requested()); + assert_eq!( + shutdown.containment_cause(), + Some("runtime controller panicked") + ); + } + + fn startup_workers_config( + http_addr: String, + ) -> (tempfile::TempDir, String, String, WorkersConfig) { + const KEY: &str = "0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80"; + + let dir = tempfile::tempdir().expect("runtime data dir"); + let data_dir = dir.path().to_str().expect("utf8 path").to_string(); + let cli = crate::harness::Cli::try_parse_from(vec![ + "sequencer".to_string(), + "run".to_string(), + "--http-addr".to_string(), + http_addr, + "--data-dir".to_string(), + data_dir.clone(), + "--eth-rpc-url".to_string(), + "http://127.0.0.1:8545".to_string(), + "--batch-submitter-private-key".to_string(), + KEY.to_string(), + ]) + .expect("parse run config"); + let crate::harness::Command::Run(run_config) = cli.command else { + panic!("expected run config"); + }; + let run_config = *run_config; + let db_path = run_config.db_path(); + let timing = run_config.protocol_timing().expect("valid timing"); + let submitter_address = + crate::commands::batch_submitter_address_from_private_key(KEY).expect("key address"); + let dumps_dir = dir.path().join("dumps"); + std::fs::create_dir(&dumps_dir).expect("create dumps dir"); + let finalized = dumps_dir.join("genesis"); + create_structured_dump(&finalized); + let mut storage = crate::storage::Storage::initialize_for_command( + &db_path, + crate::storage::LifecycleCommand::Setup, + ) + .expect("open storage"); + storage + .insert_initial_finalized_dump(&finalized, 0, 0, 0, 0) + .expect("register finalized dump"); + storage + .append_safe_inputs(0, &[], submitter_address, &timing) + .expect("initialize safe head"); + storage + .initialize_open_state(0, crate::storage::SafeInputRange::empty_at(0)) + .expect("initialize Tip"); + storage.complete_setup().expect("complete setup"); + drop(storage); + + // One identity literal feeds both the reader and the L1 bundle, so + // the fixture cannot drift the way hand-copied fields could. + let identity = crate::storage::DeploymentIdentity { + chain_id: 31337, + app_address: "0x1111111111111111111111111111111111111111" + .parse() + .expect("app address"), + input_box_address: "0x2222222222222222222222222222222222222222" + .parse() + .expect("input box address"), + app_deployment_block: 0, + batch_submitter_address: submitter_address, + fee_oracle: crate::storage::FeeOracleIdentity::Fixed { log_gas_price: 0 }, + }; + let input_reader = InputReader::from_parts( + crate::l1::reader::InputReaderConfig { + rpc_url: run_config.eth_rpc_url.clone(), + allow_insecure_rpc: false, + app_address: identity.app_address, + poll_interval: crate::commands::INPUT_READER_POLL_INTERVAL, + long_block_range_error_codes: run_config.long_block_range_error_codes.clone(), + expected_chain_id: identity.chain_id, + }, + identity.input_box_address, + identity.app_deployment_block, + db_path.clone(), + identity.batch_submitter_address, + timing, + ProcessLock::test(), + ); + let l1_config = L1Config { + identity, + eth_rpc_url: run_config.eth_rpc_url.clone(), + batch_submitter_private_key: crate::l1::SubmitterKey::new(KEY.to_string()), + allow_insecure_rpc: false, + }; + let process_lock = ProcessLock::acquire(&data_dir).expect("acquire runtime lock"); + + ( + dir, + data_dir, + db_path, + WorkersConfig { + run_config, + l1_config, + timing, + input_reader, + fee_oracle: None, + process_lock, + }, + ) + } + + #[tokio::test] + async fn occupied_http_port_fails_before_any_worker_launches() { + let occupied = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("bind occupied listener"); + let http_addr = occupied.local_addr().expect("listener address").to_string(); + let (_dir, data_dir, _db_path, workers_config) = startup_workers_config(http_addr); + + let result = PreparedRuntime::::prepare(workers_config).await; + let err = match result { + Ok(_) => panic!("occupied listener must refuse preparation"), + Err(err) => err, + }; + assert!( + matches!(&err, CommandError::Io(source) if source.kind() == std::io::ErrorKind::AddrInUse), + "expected AddrInUse, got {err:?}" + ); + // The boundary check: a worker spawned during preparation would + // retain a `RuntimeScope` clone and keep the process lock held, so + // this acquire succeeding proves zero workers launched. + ProcessLock::acquire(&data_dir) + .expect("failed preparation must release ownership with zero live workers"); + } + + #[tokio::test] + async fn launch_requires_a_fresh_admission_after_preparation() { + let (_dir, data_dir, db_path, workers_config) = + startup_workers_config("127.0.0.1:0".to_string()); + + let timing = workers_config.timing; + let prepared = PreparedRuntime::::prepare(workers_config) + .await + .expect("runtime prepares"); + let admission = crate::recovery::admit_runtime(&db_path, &timing) + .expect("the reducer admits over clean facts and returns its witness"); + let workers = prepared.launch(admission); + + workers + .finish(FirstExit::Signal(None)) + .await + .expect("workers drain cleanly"); + let _released = tokio::time::timeout(Duration::from_secs(5), async { + loop { + match ProcessLock::acquire(&data_dir) { + Ok(lock) => break lock, + Err(crate::runtime::process_lock::ProcessLockError::Locked { .. }) => { + tokio::time::sleep(Duration::from_millis(10)).await + } + Err(error) => panic!("unexpected lock acquisition failure: {error}"), + } + } + }) + .await + .expect("nested runtime work releases process ownership after clean drain"); + } + + #[tokio::test] + async fn preparation_outliving_clean_facts_cannot_launch() { + let (_dir, _data_dir, db_path, workers_config) = + startup_workers_config("127.0.0.1:0".to_string()); + let timing = workers_config.timing; + + let prepared = PreparedRuntime::::prepare(workers_config) + .await + .expect("runtime prepares"); + + // Simulate time moving behind the persisted local-progress baseline + // while preparation was in flight. The final reducer invocation must + // classify the new clock refusal before durable Live or launch. + let future_ms = i64::try_from(crate::clock::unix_now_ms()).expect("clock fits") + + i64::try_from(timing.seconds_per_block * 2_000).expect("offset fits"); + crate::storage::Storage::open_connection(&db_path) + .expect("open raw test connection") + .execute( + "UPDATE l1_safe_head SET synced_at_ms = ?1 WHERE singleton_id = 0", + [future_ms], + ) + .expect("move progress baseline into the future"); + + let error = crate::recovery::admit_runtime(&db_path, &timing) + .expect_err("aged clean decision must not admit"); + assert!(matches!(error, crate::recovery::RecoveryError::Retry(_))); + drop(prepared); + } + + #[tokio::test] + async fn contained_component_fault_surfaces_first_cause() { + // A non-worker component contains a fault: the select yields + // Contained, and finish's verdict carries the cause read back from + // the in-memory first-winner record. + let shutdown = RuntimeScope::default(); + let (mut workers, _dir) = waiting_workers(&shutdown); + + shutdown.contain_storage_invariant_failure("egress component fault: dangling dump row"); + let first_exit = workers.select_first_exit().await; + assert!(matches!(first_exit, FirstExit::Contained)); + + let err = workers + .finish(first_exit) + .await + .expect_err("contained fault must fail run"); + let CommandError::StorageInvariantViolation { cause } = &err else { + panic!("expected terminal invariant violation, got {err:?}"); + }; + assert_eq!(cause, "egress component fault: dangling dump row"); + assert_eq!(err.exit_code(), crate::commands::error::EXIT_TERMINAL); + } + + #[tokio::test] + async fn containment_overrides_an_already_selected_nonterminal_worker_exit() { + let shutdown = RuntimeScope::default(); + let (workers, _dir) = waiting_workers(&shutdown); + + shutdown.contain_storage_invariant_failure("fault raced the worker exit"); + let err = workers + .finish(FirstExit::Worker(WorkerExit::Server( + WorkerStop::StoppedUnexpectedly, + ))) + .await + .expect_err("containment must outrank the nonterminal exit"); + assert!(matches!( + err, + CommandError::StorageInvariantViolation { .. } + )); + assert_eq!(err.exit_code(), crate::commands::error::EXIT_TERMINAL); + } + + #[tokio::test] + async fn terminal_primary_worker_exit_contains_with_typed_cause() { + let shutdown = RuntimeScope::default(); + let (workers, _dir) = waiting_workers(&shutdown); + + let err = workers + .finish(FirstExit::Worker(WorkerExit::DangerDetected { + status: DangerStatus::CanonicalDivergence(7), + })) + .await + .expect_err("canonical divergence must fail run"); + let CommandError::StorageInvariantViolation { cause } = &err else { + panic!("expected terminal invariant violation, got {err:?}"); + }; + assert!( + cause.contains("terminal") && cause.contains("worker exit"), + "a terminal primary exit must surface its typed cause, got: {cause}" + ); + assert!(shutdown.is_storage_invariant_contained()); + } + + #[tokio::test] + async fn panicked_primary_contains_terminal_fault() { + let shutdown = RuntimeScope::default(); + let (mut workers, _dir) = waiting_workers(&shutdown); + workers.lane = tokio::spawn(async { panic!("lane task panicked in test") }); + + let first_exit = workers.select_first_exit().await; + let err = workers + .finish(first_exit) + .await + .expect_err("panicked worker must fail run"); + assert!(matches!( + err, + CommandError::StorageInvariantViolation { .. } + )); + assert!(shutdown.is_storage_invariant_contained()); + } + + #[tokio::test] + async fn recovery_primary_uses_ordinary_shutdown_without_containment() { + let shutdown = RuntimeScope::default(); + let (workers, _dir) = waiting_workers(&shutdown); + + let err = workers + .finish(FirstExit::Worker(WorkerExit::DangerDetected { + status: DangerStatus::TipInDanger(3), + })) + .await + .expect_err("danger exit must fail run"); + assert!(matches!(err, CommandError::Worker(_))); + assert!(!shutdown.is_storage_invariant_contained()); + } + + #[tokio::test] + async fn transient_primary_uses_ordinary_shutdown_without_containment() { + let shutdown = RuntimeScope::default(); + let (workers, _dir) = waiting_workers(&shutdown); + + let err = workers + .finish(FirstExit::Worker(WorkerExit::Server( + WorkerStop::StoppedUnexpectedly, + ))) + .await + .expect_err("unexpected server stop must fail run"); + assert!(matches!(err, CommandError::Worker(_))); + assert!(!shutdown.is_storage_invariant_contained()); + } + + /// The oracle's cleanup entry exists only when the worker does, and the + /// primary-removal expect relies on that coupling — pin the `Some` limb, + /// which no production-path test exercises (every fixture is fixed-mode). + #[tokio::test] + async fn fee_oracle_primary_exit_drains_through_its_conditional_entry() { + let shutdown = RuntimeScope::default(); + let (mut workers, _dir) = waiting_workers(&shutdown); + workers.fee_oracle = Some(tokio::spawn({ + let shutdown = shutdown.clone(); + async move { + shutdown.wait_for_shutdown().await; + Ok(()) + } + })); + + let err = workers + .finish(FirstExit::Worker(WorkerExit::FeeOracle( + WorkerStop::StoppedUnexpectedly, + ))) + .await + .expect_err("unexpected oracle stop must fail run"); + assert!(matches!( + err, + CommandError::Worker(WorkerExit::FeeOracle(WorkerStop::StoppedUnexpectedly)) + )); + assert!(!shutdown.is_storage_invariant_contained()); + } + + #[tokio::test] + async fn terminal_cleanup_error_overrides_nonterminal_primary_exit() { + let shutdown = RuntimeScope::default(); + let (mut workers, _dir) = waiting_workers(&shutdown); + workers.reader = tokio::spawn(async { + Err(InputReaderError::StorageTaskPanicked { + operation: "reading corrupt state during cleanup", + }) + }); + + let err = workers + .finish(FirstExit::Worker(WorkerExit::Server( + WorkerStop::StoppedUnexpectedly, + ))) + .await + .expect_err("terminal cleanup error must outrank nonterminal primary"); + let CommandError::StorageInvariantViolation { cause } = &err else { + panic!("expected terminal invariant violation, got {err:?}"); + }; + assert!( + cause.contains("cleanup-time") + && cause.contains("reading corrupt state during cleanup"), + "a cleanup-time terminal exit must surface its typed cause, got: {cause}" + ); + assert!(shutdown.is_storage_invariant_contained()); + } + + #[tokio::test] + async fn terminal_cleanup_is_observed_while_another_component_is_still_draining() { + let shutdown = RuntimeScope::default(); + let (mut workers, _dir) = waiting_workers(&shutdown); + let (release_server, server_released) = tokio::sync::oneshot::channel(); + workers.server = tokio::spawn(async move { + let _ = server_released.await; + Ok(()) + }); + workers.reader = tokio::spawn(async { + Err(InputReaderError::StorageTaskPanicked { + operation: "terminal cleanup behind a draining server", + }) + }); + + let finish = tokio::spawn(workers.finish(FirstExit::Worker(WorkerExit::Lane( + WorkerStop::StoppedUnexpectedly, + )))); + tokio::time::timeout(Duration::from_secs(1), async { + while !shutdown.is_storage_invariant_contained() { + tokio::task::yield_now().await; + } + }) + .await + .expect("a draining earlier component must not hide a terminal cleanup exit"); + + release_server.send(()).expect("release server drain"); + let err = finish + .await + .expect("finish task") + .expect_err("terminal cleanup must fail the run"); + let CommandError::StorageInvariantViolation { cause } = err else { + panic!("expected terminal invariant violation"); + }; + assert!(cause.contains("terminal cleanup behind a draining server")); + } + + #[tokio::test] + async fn signal_cleanup_contains_terminal_fault() { + let shutdown = RuntimeScope::default(); + let (mut workers, _dir) = waiting_workers(&shutdown); + workers.reader = tokio::spawn(async { + Err(InputReaderError::StorageTaskPanicked { + operation: "reading corrupt state during signal drain", + }) + }); + + let err = workers + .finish(FirstExit::Signal(None)) + .await + .expect_err("terminal fault during signal drain must fail run"); + let CommandError::StorageInvariantViolation { cause } = &err else { + panic!("expected terminal invariant violation, got {err:?}"); + }; + assert!(cause.contains("shutdown-time"), "got: {cause}"); + assert!(shutdown.is_storage_invariant_contained()); + } + + use crate::commands::test_support::create_structured_dump; + use crate::storage::Storage; + use crate::storage::test_helpers::temp_db; + + #[test] + fn detector_inner_error_maps_to_source_variant() { + let result: DetectorJoinResult = Ok(Err(DangerDetectorError::Join("boom".into()))); + assert!(matches!( + FirstExit::detector(result), + FirstExit::Worker(WorkerExit::DangerDetector(WorkerStop::Source(_))) + )); + } +} diff --git a/sequencer/src/runtime/setup_fill.rs b/sequencer/src/commands/setup/fill.rs similarity index 63% rename from sequencer/src/runtime/setup_fill.rs rename to sequencer/src/commands/setup/fill.rs index fa1df840..19877cd0 100644 --- a/sequencer/src/runtime/setup_fill.rs +++ b/sequencer/src/commands/setup/fill.rs @@ -15,10 +15,10 @@ //! never by a runtime worker — hence their own module, distinct from the //! worker lifecycle in [`super::workers`]. +use crate::commands::error::{CommandError, SetupRecoveryError}; use crate::ingress::inclusion_lane::dump_info::{ self, CreateDumpDirError, create_dump_dir_with_info, }; -use crate::runtime::error::{RunError, SetupRecoveryError}; use sequencer_core::application::Application; /// Register the genesis application state as the finalized snapshot. Called @@ -33,10 +33,24 @@ pub(crate) fn register_genesis_finalized_snapshot( initial_app: A, storage: &mut crate::storage::Storage, dumps_dir: &std::path::Path, -) -> Result<(), RunError> { +) -> Result<(), CommandError> { if storage.finalized_dump()?.is_some() { return Ok(()); } + // The violator here is a foreign `Application` impl supplied by the app + // crate, so a nonzero genesis boundary is a typed refusal with a + // diagnosis, not a panic across the crate boundary. It still runs + // first, before any write; nothing to unwind. + let genesis_count = initial_app.executed_input_count().get(); + if genesis_count != 0 { + return Err(CommandError::AppBootstrap( + sequencer_core::application::AppError::Internal { + reason: format!( + "a genesis application must start at executed_input_count = 0, got {genesis_count}" + ), + }, + )); + } let nanos = std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) .map(|d| d.as_nanos()) @@ -53,10 +67,10 @@ pub(crate) fn register_genesis_finalized_snapshot( }, ) .map_err(|err| match err { - CreateDumpDirError::App(e) => RunError::from(e), - CreateDumpDirError::Io(e) => RunError::from(e), + CreateDumpDirError::App(e) => CommandError::from(e), + CreateDumpDirError::Io(e) => CommandError::from(e), })?; - storage.insert_finalized_dump(&genesis_dir, 0, 0)?; + storage.insert_initial_finalized_dump(&genesis_dir, 0, 0, 0, 0)?; Ok(()) } @@ -67,9 +81,11 @@ pub(crate) fn register_genesis_finalized_snapshot( /// startup expects: a finalized snapshot of `S'`, a batch tree rooted at `N'`, /// and a replay cursor past every `≤ C` direct (already executed inside `S'`). /// -/// Crash-before-marker re-entry is **fail-loud**, not blindly idempotent. Only a +/// Pre-completion re-entry is **fail-loud**, not blindly idempotent. Only a /// *completed* fill (finalized snapshot present — the last write) re-runs as a -/// safe no-op. Any other existing root tip is a crashed mid-fill and is refused: +/// safe no-op: its atomically bound snapshot and history base remain +/// authoritative even if the retry's newer fold reached a later count. Any +/// other existing root tip is a crashed mid-fill and is refused: /// * a *different* `N'` (a different checkpoint, or the same one after `C` /// advanced with more accepted `(B, C]` **batches**) would re-anchor the /// tree while leaving the old root tip in place, silently breaking I16 — @@ -96,7 +112,12 @@ pub(crate) fn fill_recovery_state( stop_block: u64, storage: &mut crate::storage::Storage, dumps_dir: &std::path::Path, -) -> Result<(), RunError> { +) -> Result<(), CommandError> { + // `K`: the absolute application boundary recovered by the canonical fold. + // It is deliberately independent of the replacement DB's physical replay + // cursor, which includes cursor-padding rows that must not execute again. + let base_executed_input_count = recovered_app.executed_input_count().get(); + // Re-entry guard (rationale + the strict one-shot model are in the // docstring). The tip is opened before the snapshot, so an existing root tip // means a prior attempt: only a *completed* fill (finalized snapshot present) @@ -124,7 +145,7 @@ pub(crate) fn fill_recovery_state( // proceeds below (re-anchoring + opening the tip completes it). But a // finalized snapshot with *no* root tip is residue from a different // deployment mode: a plain `setup` that wrote the genesis snapshot and - // crashed before its marker. A completed cockroach fill always has both + // crashed before completion. A completed cockroach fill always has both // (caught above), so reaching here with a snapshot means recovery is running // over an un-wiped data dir — silently keeping it would mark setup complete // over genesis instead of the folded `(S', N')`. Refuse (same fail-loud @@ -150,6 +171,11 @@ pub(crate) fn fill_recovery_state( // 3. The finalized snapshot's replay cursor = the global valid replay head // AFTER step 2's sequencing. let head = storage.valid_ordered_l2_tx_head()?; + // The root's exclusive safe-input cursor is a separate durable floor. Its + // padding rows may later disappear from the valid view when standard + // recovery invalidates this root, but inputs already represented by S' + // must never become drainable again. + let base_safe_input_index = storage.next_undrained_safe_input_index()?; // 4. Register S' as the finalized snapshot at block C (file-first). Unique // per attempt so a crash before the DB row leaves only a swept orphan. let nanos = std::time::SystemTime::now() @@ -163,32 +189,411 @@ pub(crate) fn fill_recovery_state( &dump_info::DumpInfo::at_recovery(resume_nonce, head, stop_block), ) .map_err(|err| match err { - CreateDumpDirError::App(e) => RunError::from(e), - CreateDumpDirError::Io(e) => RunError::from(e), + CreateDumpDirError::App(e) => CommandError::from(e), + CreateDumpDirError::Io(e) => CommandError::from(e), })?; - storage.insert_finalized_dump(&recovery_dir, stop_block, head)?; + storage.insert_initial_finalized_dump( + &recovery_dir, + stop_block, + head, + base_executed_input_count, + base_safe_input_index, + )?; Ok(()) } #[cfg(test)] mod tests { use super::*; + use crate::commands::test_support::SweepTestApp; use crate::ingress::inclusion_lane::{InclusionLane, InclusionLaneConfig, InclusionLaneError}; - use crate::runtime::shutdown::ShutdownSignal; - use crate::runtime::test_support::SweepTestApp; - use crate::storage::Storage; - use crate::storage::test_helpers::temp_db; + use crate::runtime::shutdown::RuntimeScope; + use crate::storage::test_helpers::{pin_test_deployment_identity, temp_db}; + use crate::storage::{LifecycleCommand, Storage}; use alloy_primitives::{Address, U256}; use app_core::application::{WalletApp, WalletConfig}; + use sequencer_core::application::{ + AppError, AppOutputs, ApplicationProgress, ApplyInputCapability, InvalidReason, + ProgressCommitCapability, + }; + use sequencer_core::history::ExecutedInputCount; + use sequencer_core::l2_tx::{DirectInput, SequencedL2Tx, ValidUserOp}; + use sequencer_core::user_op::UserOp; + use std::path::Path; use std::time::Duration; + #[derive(Clone)] + struct CountedSweepTestApp(ApplicationProgress); + + impl CountedSweepTestApp { + fn new(executed_input_count: u64) -> Self { + Self(ApplicationProgress::new( + ExecutedInputCount::new(executed_input_count), + 0, + )) + } + } + + impl Application for CountedSweepTestApp { + const MAX_METHOD_PAYLOAD_BYTES: usize = 0; + + fn validate_user_op( + &self, + _sender: Address, + _user_op: &UserOp, + _current_fee: u16, + ) -> Result<(), InvalidReason> { + Ok(()) + } + + fn apply_valid_user_op( + &mut self, + _capability: ApplyInputCapability<'_>, + _user_op: &ValidUserOp, + _safe_block: u64, + ) -> Result { + unreachable!("not used by setup-fill tests") + } + + fn apply_direct_input( + &mut self, + _capability: ApplyInputCapability<'_>, + _input: &DirectInput, + ) -> Result { + Ok(Vec::new()) + } + + fn execution_progress(&self) -> &ApplicationProgress { + &self.0 + } + + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + &mut self.0 + } + + fn from_dump(prefix: &Path) -> Result { + let bytes = std::fs::read(prefix.join("state"))?; + let bytes: [u8; 16] = bytes.try_into().map_err(|_| AppError::Internal { + reason: "invalid counted test dump".to_string(), + })?; + let count = u64::from_le_bytes(bytes[..8].try_into().expect("eight-byte count")); + let safe_block = + u64::from_le_bytes(bytes[8..].try_into().expect("eight-byte safe block")); + Ok(Self(ApplicationProgress::new( + ExecutedInputCount::new(count), + safe_block, + ))) + } + + fn create_dump(&self, prefix: &Path) -> Result<(), AppError> { + std::fs::create_dir(prefix)?; + let mut bytes = Vec::with_capacity(16); + bytes.extend_from_slice(&self.0.executed_input_count().get().to_le_bytes()); + bytes.extend_from_slice(&self.0.last_executed_safe_block().to_le_bytes()); + std::fs::write(prefix.join("state"), bytes)?; + Ok(()) + } + + fn delete_dump(prefix: &Path) -> Result<(), AppError> { + ::delete_dump(prefix) + } + + fn state_file_in_dump(prefix: &Path) -> std::path::PathBuf { + ::state_file_in_dump(prefix) + } + } + + #[test] + fn recovery_binds_absolute_application_base_not_physical_cursor() { + use crate::storage::StoredSafeInput; + use crate::storage::test_helpers::default_protocol_timing; + + let db = temp_db("fill-recovery-history-base"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Rebuild) + .expect("initialize rebuild"); + let dumps_dir = tempfile::tempdir().expect("dumps dir"); + let submitter = Address::repeat_byte(0x99); + let direct = Address::repeat_byte(0x22); + storage + .append_safe_inputs( + 100, + &[ + StoredSafeInput { + sender: direct, + payload: vec![0x01], + block_number: 20, + }, + StoredSafeInput { + sender: direct, + payload: vec![0x02], + block_number: 30, + }, + ], + submitter, + &default_protocol_timing(), + ) + .expect("sync recovered inputs"); + + fill_recovery_state( + CountedSweepTestApp::new(41), + 3, + 100, + &mut storage, + dumps_dir.path(), + ) + .expect("fill recovered state"); + + assert_eq!( + storage + .history_state() + .expect("history") + .base_executed_input_count, + Some(41), + "K comes from the recovered Application state" + ); + assert_eq!( + storage + .history_state() + .expect("history") + .base_safe_input_index, + Some(2), + "the recovery root's exclusive safe-input cursor becomes the durable drain floor" + ); + let finalized = storage + .finalized_dump() + .expect("read finalized") + .expect("finalized snapshot"); + assert_eq!( + finalized.l2_tx_index, 2, + "the replacement DB cursor can differ from absolute application count K" + ); + assert_eq!( + CountedSweepTestApp::from_dump(&dump_info::app_prefix(&finalized.dump.prefix)) + .expect("reload counted snapshot") + .executed_input_count() + .get(), + 41, + "the snapshot bytes and durable K establish the same application boundary" + ); + + // Model a retry whose re-sync/fold advanced through more direct inputs + // without changing N'. The completed snapshot/base pair is already the + // durable boundary; the later fold must not reinterpret this era. + fill_recovery_state( + CountedSweepTestApp::new(42), + 3, + 100, + &mut storage, + dumps_dir.path(), + ) + .expect("a completed fill remains authoritative across a later fold"); + assert_eq!( + storage + .history_state() + .expect("preserved history") + .base_executed_input_count, + Some(41), + "a completed retry preserves the snapshot-bound K" + ); + assert_eq!( + storage + .history_state() + .expect("preserved history") + .base_safe_input_index, + Some(2), + "a completed retry preserves the snapshot-bound drain floor" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn standard_recovery_never_redrains_below_cockroach_floor() { + use crate::storage::StoredSafeInput; + use crate::storage::test_helpers::default_protocol_timing; + + let db = temp_db("cockroach-drain-floor"); + let dumps_dir = tempfile::tempdir().expect("dumps dir"); + let timing = default_protocol_timing(); + let submitter = Address::repeat_byte(0x99); + let direct = Address::repeat_byte(0x22); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Rebuild) + .expect("initialize rebuild"); + pin_test_deployment_identity(&mut storage, submitter); + storage + .append_safe_inputs( + 100, + &[ + StoredSafeInput { + sender: direct, + payload: vec![0x01], + block_number: 20, + }, + StoredSafeInput { + sender: direct, + payload: vec![0x02], + block_number: 30, + }, + ], + submitter, + &timing, + ) + .expect("sync recovered inputs"); + + // S' already includes these two directs. The root sequences them only + // as physical replay padding, and binds their exclusive cursor as the + // durable floor beside the snapshot. + fill_recovery_state( + CountedSweepTestApp::new(41), + 3, + 100, + &mut storage, + dumps_dir.path(), + ) + .expect("fill recovered state"); + storage.complete_setup().expect("complete rebuild"); + assert_eq!( + storage + .history_state() + .expect("history") + .base_safe_input_index, + Some(2) + ); + + // Age the recovery root into standard recovery. Invalidating it removes + // its padding rows from the valid view; the replacement Tip must still + // begin at the durable floor rather than re-sequencing indices 0 and 1. + storage + .append_safe_inputs( + 1_500, + &[StoredSafeInput { + sender: direct, + payload: vec![0x03], + block_number: 1_400, + }], + submitter, + &timing, + ) + .expect("advance safe head with one post-floor direct"); + assert_eq!( + storage.recover_post_flush(1_200).expect("recover root"), + vec![0] + ); + assert_eq!( + storage + .next_undrained_safe_input_index() + .expect("post-recovery cursor"), + 3 + ); + let replay = storage + .ordered_l2_txs_page_from(0, 16) + .expect("valid replay"); + assert_eq!( + replay.len(), + 1, + "only the post-floor direct belongs to replacement history" + ); + let row = &replay[0]; + assert_eq!( + row.executed_input_offset, + Some(ExecutedInputCount::new(41)), + "the first post-floor direct must reuse the recovered application boundary K" + ); + match &row.tx { + SequencedL2Tx::Direct(input) => assert_eq!(input.payload, [0x03]), + SequencedL2Tx::UserOp(_) => panic!("expected the post-floor direct"), + } + drop(storage); + + // Restart the real lane from S'. Catch-up must execute only the + // post-floor direct at offset 41, advancing the application to 42. If + // the invalidated padding were re-sequenced, the count would be larger. + // Force an empty batch close so the post-catch-up state is visible. + let storage = Storage::open(db.path.as_str()).expect("reopen for lane"); + let config = InclusionLaneConfig { + batch_submitter_address: submitter, + dumps_dir: dumps_dir.path().to_path_buf(), + max_user_ops_per_chunk: 16, + safe_input_buffer_capacity: 16, + max_batch_open: Duration::from_millis(10), + idle_poll_interval: Duration::from_millis(2), + frontier_min_interval: Duration::ZERO, + }; + let shutdown = RuntimeScope::default(); + let (_tx, handle) = + InclusionLane::::start(16, shutdown.clone(), storage, config); + + let advanced_once = wait_until(Duration::from_secs(5), || { + let mut observer = Storage::open(db.path.as_str()).expect("open observer"); + let Some(pending) = observer.latest_pending_dump().expect("read pending") else { + return false; + }; + CountedSweepTestApp::from_dump(&dump_info::app_prefix(&pending.dump.prefix)) + .expect("load post-catch-up snapshot") + .executed_input_count() + .get() + == 42 + }) + .await; + assert!( + advanced_once, + "lane catch-up must execute the post-floor direct exactly once" + ); + + shutdown_lane(&shutdown, handle).await; + } + + #[test] + fn plain_setup_refuses_a_nonzero_genesis_application_boundary() { + let db = temp_db("genesis-history-base"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Setup) + .expect("initialize setup"); + let dumps_dir = tempfile::tempdir().expect("dumps dir"); + + let result = register_genesis_finalized_snapshot( + CountedSweepTestApp::new(1), + &mut storage, + dumps_dir.path(), + ); + assert!( + matches!( + result, + Err(CommandError::AppBootstrap( + sequencer_core::application::AppError::Internal { .. } + )) + ), + "genesis must begin at application count zero, got: {result:?}" + ); + assert!(storage.finalized_dump().expect("read finalized").is_none()); + assert_eq!( + storage + .history_state() + .expect("history") + .base_executed_input_count, + Some(0) + ); + assert_eq!( + storage + .history_state() + .expect("history") + .base_safe_input_index, + Some(0) + ); + } + #[test] fn fill_recovery_state_roots_tree_at_n_prime_and_skips_pre_executed_directs() { use crate::storage::StoredSafeInput; use crate::storage::test_helpers::default_protocol_timing; let db = temp_db("fill-recovery"); - let mut storage = Storage::open(db.path.as_str()).expect("open"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Rebuild) + .expect("initialize rebuild"); let dumps_dir = tempfile::tempdir().expect("dumps dir"); let submitter = alloy_primitives::Address::repeat_byte(0x99); let direct = alloy_primitives::Address::repeat_byte(0x22); @@ -255,7 +660,7 @@ mod tests { "catch-up starts past the pre-executed (≤C) directs" ); - // Idempotent: a re-run (crash before the setup marker) is a no-op, not a + // Idempotent: a re-run (crash before setup completion) is a no-op, not a // duplicate-insert error. fill_recovery_state(SweepTestApp, n_prime, 100, &mut storage, dumps_dir.path()) .expect("re-run is idempotent"); @@ -264,7 +669,7 @@ mod tests { #[test] fn fill_recovery_state_leaves_post_c_directs_undrained() { - // P1: the post-flush resync runs to the live safe head H1, normally > the + // The post-flush resync runs to the live safe head H1, normally > the // checkpoint stop block C. The fold folds only `<= C` into S'; directs in // (C, H1] must stay UNDRAINED so run leads + executes them exactly once. // Draining them here (the old behavior) would skip them on catch-up while @@ -273,7 +678,9 @@ mod tests { use crate::storage::test_helpers::default_protocol_timing; let db = temp_db("fill-recovery-post-c"); - let mut storage = Storage::open(db.path.as_str()).expect("open"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Rebuild) + .expect("initialize rebuild"); let dumps_dir = tempfile::tempdir().expect("dumps dir"); let submitter = alloy_primitives::Address::repeat_byte(0x99); let direct = alloy_primitives::Address::repeat_byte(0x22); @@ -338,7 +745,9 @@ mod tests { use crate::storage::test_helpers::default_protocol_timing; let db = temp_db("fill-recovery-boundary"); - let mut storage = Storage::open(db.path.as_str()).expect("open"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Rebuild) + .expect("initialize rebuild"); let dumps_dir = tempfile::tempdir().expect("dumps dir"); let submitter = alloy_primitives::Address::repeat_byte(0x99); let direct = alloy_primitives::Address::repeat_byte(0x22); @@ -386,14 +795,16 @@ mod tests { #[test] fn fill_recovery_state_refuses_re_run_with_a_different_nonce() { - // B1: a re-run with a *different* resume nonce (e.g. a different + // A re-run with a *different* resume nonce (e.g. a different // checkpoint, or the same one after C advanced) would move the anchor // while leaving the old root tip — a silent I16 break. It must fail loud. use crate::storage::StoredSafeInput; use crate::storage::test_helpers::default_protocol_timing; let db = temp_db("fill-recovery-nonce-mismatch"); - let mut storage = Storage::open(db.path.as_str()).expect("open"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Rebuild) + .expect("initialize rebuild"); let dumps_dir = tempfile::tempdir().expect("dumps dir"); let submitter = alloy_primitives::Address::repeat_byte(0x99); let timing = default_protocol_timing(); @@ -417,7 +828,7 @@ mod tests { assert!( matches!( err, - RunError::Bootstrap(crate::runtime::error::BootstrapError::SetupRecovery( + CommandError::Bootstrap(crate::commands::error::BootstrapError::SetupRecovery( SetupRecoveryError::PartialRecoveryMismatch { existing_root_nonce: 3, requested_nonce: 5, @@ -430,7 +841,7 @@ mod tests { #[test] fn fill_recovery_state_refuses_incomplete_same_nonce_re_run() { - // P1: a crash between opening the recovery root tip (fill step 2) and + // A crash between opening the recovery root tip (fill step 2) and // writing the finalized snapshot (step 4) leaves "tip exists, no // finalized". A same-N' re-run must NOT resume — directs that landed // since would be left unsequenced, leaving the snapshot cursor behind @@ -445,7 +856,9 @@ mod tests { use crate::storage::test_helpers::default_protocol_timing; let db = temp_db("fill-recovery-incomplete"); - let mut storage = Storage::open(db.path.as_str()).expect("open"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Rebuild) + .expect("initialize rebuild"); let dumps_dir = tempfile::tempdir().expect("dumps dir"); let submitter = alloy_primitives::Address::repeat_byte(0x99); let direct = alloy_primitives::Address::repeat_byte(0x22); @@ -496,7 +909,7 @@ mod tests { assert!( matches!( err, - RunError::Bootstrap(crate::runtime::error::BootstrapError::SetupRecovery( + CommandError::Bootstrap(crate::commands::error::BootstrapError::SetupRecovery( SetupRecoveryError::PartialRecoveryIncomplete { root_nonce: 3 } )) ), @@ -506,14 +919,16 @@ mod tests { #[test] fn fill_recovery_state_refuses_over_residual_finalized_snapshot() { - // P2: `setup --recovery` over an un-wiped data dir left by a plain + // `setup --recovery` over an un-wiped data dir left by a plain // `setup` that wrote the genesis finalized snapshot and crashed before - // its marker (finalized snapshot present, NO root tip). A completed + // completion (finalized snapshot present, NO root tip). A completed // cockroach fill always has both, so this residue must fail loud — // silently keeping it would mark setup complete over genesis instead of // the folded `(S', N')`. let db = temp_db("fill-recovery-residue"); - let mut storage = Storage::open(db.path.as_str()).expect("open"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Setup) + .expect("initialize plain setup residue"); let dumps_dir = tempfile::tempdir().expect("dumps dir"); // Plain-setup residue: genesis finalized snapshot, no root tip. @@ -529,7 +944,7 @@ mod tests { assert!( matches!( err, - RunError::Bootstrap(crate::runtime::error::BootstrapError::SetupRecovery( + CommandError::Bootstrap(crate::commands::error::BootstrapError::SetupRecovery( SetupRecoveryError::RecoveryOverResidualSnapshot { existing_finalized_block: 0, } @@ -607,7 +1022,10 @@ mod tests { }, ]; { - let mut storage = Storage::open(db.path.as_str()).expect("open"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Rebuild) + .expect("initialize rebuild"); + pin_test_deployment_identity(&mut storage, submitter); storage .append_safe_inputs(150, &inputs, submitter, &timing) .expect("sync to H1 = 150"); @@ -648,7 +1066,7 @@ mod tests { // input; the short `max_batch_open` forces a batch-close snapshot. let storage = Storage::open(db.path.as_str()).expect("reopen for lane"); let config = InclusionLaneConfig { - batch_submitter_address: Address::repeat_byte(0xff), + batch_submitter_address: submitter, dumps_dir: dumps_dir.path().to_path_buf(), max_user_ops_per_chunk: 16, safe_input_buffer_capacity: 16, @@ -656,7 +1074,7 @@ mod tests { idle_poll_interval: Duration::from_millis(2), frontier_min_interval: Duration::ZERO, }; - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let (_tx, handle) = InclusionLane::::start(128, shutdown.clone(), storage, config); @@ -719,7 +1137,7 @@ mod tests { } async fn shutdown_lane( - shutdown: &ShutdownSignal, + shutdown: &RuntimeScope, handle: tokio::task::JoinHandle>, ) { shutdown.request_shutdown(); diff --git a/sequencer/src/runtime/setup.rs b/sequencer/src/commands/setup/mod.rs similarity index 76% rename from sequencer/src/runtime/setup.rs rename to sequencer/src/commands/setup/mod.rs index 87463cb7..67106dbe 100644 --- a/sequencer/src/runtime/setup.rs +++ b/sequencer/src/commands/setup/mod.rs @@ -4,19 +4,22 @@ //! The `setup` subcommand: establish everything timeless //! and pin it in the DB, then mark setup complete so `run` will boot. //! -//! Steps, in order — each individually idempotent so a crashed `setup` -//! re-runs cleanly, and the marker written *last* is the single -//! linearization point for "A finished": +//! Steps, in order. The storage phases are re-entry-safe (a crashed +//! attempt's retry begins fresh over its idempotent residue), and the final +//! transaction is the single linearization point for "A finished": //! //! 1. Validate protocol timing; create the data dir + `dumps/`. //! 2. Require L1: discover the InputBox address + app deployment block from the app //! contract, validate the RPC chain id. (No cached-identity fallback — //! this is first-boot; an unreachable L1 is a retryable refusal.) -//! 3. Pin the deployment identity (chain id, app address, InputBox address, -//! app deployment block, batch-submitter **address** — `setup` never signs). +//! 3. Resolve the fee source, pin the complete deployment identity (chain id, +//! app address, InputBox address, app deployment block, batch-submitter +//! **address**, and fee-oracle identity), and persist the first price. +//! `setup` never signs. //! 4. Initial L1 sync: read all direct inputs up to the current safe head. -//! 5. Register the genesis application state as the finalized snapshot. -//! 6. Write the `setup_complete` marker. +//! 5. For plain setup, construct and register the genesis application state as +//! the finalized snapshot. Recovery supplies its state from the checkpoint. +//! 6. Commit the `setup_complete` fact. //! //! `setup` is L1-read-only: it takes the batch-submitter address (not the //! key) and does no L1 writes. @@ -25,20 +28,22 @@ use alloy_primitives::Address; use sequencer_core::application::Application; use sequencer_core::scheduler::{FoldInput, SchedulerConfig, fold_replay}; -use super::config::{FeeOracleMode, SetupConfig}; -use super::{ - BootstrapError, IdentityError, InputReaderExit, RunError, SetupRecoveryError, SetupRefuse, - WorkerExit, ensure_deployment_identity, setup_fill, validate_rpc_chain_id, +pub(crate) mod fill; + +use super::{ensure_deployment_identity, validate_rpc_chain_id}; +use crate::commands::config::{FeeOracleMode, SetupConfig}; +use crate::commands::error::{ + BootstrapError, CommandError, IdentityError, SetupRecoveryError, SetupRefuse, WorkerExit, }; use crate::ingress::inclusion_lane::dump_info; -use crate::l1::provider::VerifiedSignerProviderError; use crate::l1::reader::{InputReader, InputReaderConfig, InputReaderError}; use crate::recovery::{MempoolFlusher, assert_resync_caught_up}; use crate::storage::{self, DeploymentIdentity, FeeOracleIdentity}; -pub async fn setup(config: SetupConfig, genesis_app: A) -> Result<(), RunError> +pub async fn setup(config: SetupConfig, genesis_app: F) -> Result<(), CommandError> where A: Application + 'static, + F: FnOnce() -> A, { // Cross-field config validation (recovery vs the recovery-only args). A // misconfig is operator error — terminal, before any filesystem touch. @@ -47,31 +52,47 @@ where .map_err(|message| SetupRecoveryError::InvalidConfig { message })?; std::fs::create_dir_all(&config.data_dir)?; - let dumps_dir = std::path::Path::new(&config.data_dir).join("dumps"); - std::fs::create_dir_all(&dumps_dir)?; - let db_path = config.db_path(); - let timing = config.timing.protocol_timing()?; + // Exclusive process ownership before any read or mutation: setup rewrites + // deployment state and must never run beside a live sequencer (or a + // second setup) on the same data dir. + let process_lock = crate::runtime::process_lock::ProcessLock::acquire(&config.data_dir)?; - // The `setup_complete` marker gates re-invocation differently per mode: - // * plain `setup` is idempotent — a re-run on a complete DB is a no-op - // success (a half-finished setup has no marker and re-runs below); - // * `setup --recovery` is a strict one-shot on a freshly-wiped DB — a - // complete DB means recovery already ran (or this is a live deployment), - // and re-pointing it at a different checkpoint would strand its state, - // so refuse (terminal). A crash-*before*-marker recovery has no marker; - // its fill is not blindly idempotent either — the partial/residue cases - // are handled fail-loud by `setup_fill::fill_recovery_state` (see there). + let db_path = config.db_path(); + config.timing.protocol_timing()?; + let command = if config.recovery { + storage::LifecycleCommand::Rebuild + } else { + storage::LifecycleCommand::Setup + }; + if let SetupAdmission::AlreadyComplete = + admit_setup_lifecycle(&db_path, command, config.recovery)? { - let storage = storage::Storage::open(&db_path)?; - if storage.is_setup_complete()? { - if config.recovery { - return Err(SetupRecoveryError::AlreadySetUp.into()); - } - tracing::info!(data_dir = %config.data_dir, "setup already complete — nothing to do"); - return Ok(()); - } + tracing::info!(data_dir = %config.data_dir, "setup already complete — nothing to do"); + return Ok(()); } + let result = setup_admitted(config, genesis_app, process_lock.clone()).await; + settle_setup_lifecycle(&db_path, command, &result)?; + result +} + +async fn setup_admitted( + config: SetupConfig, + genesis_app: F, + process_lock: crate::runtime::process_lock::ProcessLock, +) -> Result<(), CommandError> +where + A: Application + 'static, + F: FnOnce() -> A, +{ + let db_path = config.db_path(); + let timing = config.timing.protocol_timing()?; + + // Do not create auxiliary data-dir entries before durable lifecycle + // admission. The lock file itself is the unavoidable ownership anchor. + let dumps_dir = std::path::Path::new(&config.data_dir).join("dumps"); + std::fs::create_dir_all(&dumps_dir)?; + // ── L1 discovery (required) ────────────────────────────── let input_reader_config = InputReaderConfig { rpc_url: config.eth_rpc_url.clone(), @@ -88,6 +109,7 @@ where input_reader_config, config.batch_submitter_address, timing, + process_lock.clone(), ) .await { @@ -99,8 +121,8 @@ where return Err(IdentityError::FirstBootRequiresL1.into()); } Err(source) => { - return Err(RunError::Worker(WorkerExit::InputReader( - InputReaderExit::Source(source), + return Err(CommandError::Worker(WorkerExit::InputReader( + crate::commands::error::WorkerStop::Source(source), ))); } }; @@ -133,23 +155,22 @@ where let prepared_fee_oracle = match fee_oracle { FeeOracleMode::Fixed { log_gas_price } => PreparedFeeOracle::Fixed { log_gas_price }, FeeOracleMode::Uniswap(uniswap) => { - let provider = crate::l1::provider::create_provider( + // Setup requires L1: transient and misconfig both abort, before + // anything pins (one connect/classify home). + let (provider, token) = crate::l1::fee_oracle::connect_uniswap( &config.eth_rpc_url, config.allow_insecure_rpc, + uniswap, ) - .map_err(|message| BootstrapError::FeeOracleMisconfig { message })?; - let token = - crate::l1::fee_oracle::UniswapV3PriceSource::connect(provider.clone(), uniswap) - .await - .map_err(|error| { - let (transient, message) = - crate::l1::fee_oracle::uniswap::bootstrap_price_source_error(error); - if transient { - BootstrapError::FeeOracleTransient { message } - } else { - BootstrapError::FeeOracleMisconfig { message } - } - })?; + .await + .map_err(|error| match error { + crate::l1::fee_oracle::UniswapConnectError::Transient { message, .. } => { + BootstrapError::FeeOracleTransient { message } + } + crate::l1::fee_oracle::UniswapConnectError::Misconfig(message) => { + BootstrapError::FeeOracleMisconfig { message } + } + })?; PreparedFeeOracle::Uniswap { identity: FeeOracleIdentity::Uniswap { weth: uniswap.weth, @@ -195,15 +216,13 @@ where PreparedFeeOracle::Uniswap { provider, token, .. } => { - let max_price_age_ms = timing.l1_read_stale_after_secs().saturating_mul(1000); - let oracle = crate::l1::fee_oracle::FeeOracle::new( + crate::l1::fee_oracle::persist_first_price( db_path.clone(), - crate::l1::fee_oracle::FeeOracle::DEFAULT_POLL_INTERVAL, - max_price_age_ms, provider, - Box::new(token), - ); - oracle.refresh_once().await?; + token, + process_lock.clone(), + ) + .await?; } } @@ -211,8 +230,8 @@ where // A checkpoint promotion cannot predate the application's deployment // block (the scan genesis — no input exists before it). // `B = 0` is the genesis bootstrap (no checkpoint) and is always valid. - // (PR3 detects only; loading a non-genesis checkpoint machine and the - // `A < B` check are `setup --recovery` / PR5.) + // (plain setup detects only; loading a non-genesis checkpoint machine and + // the `A < B` check are `setup --recovery`'s job.) if config.checkpoint_block != 0 && config.checkpoint_block < input_reader.app_deployment_block() { return Err(BootstrapError::CheckpointBeforeAppDeployment { @@ -225,9 +244,9 @@ where // ── Initial L1 sync ────────────────────────────────────── // One pass reads every direct input up to the current safe head into // `safe_inputs` + `safe_accepted_batches` and persists `l1_safe_head`. - // Idempotent: a retried setup resumes from the persisted safe head. A - // transient sync failure leaves the marker unwritten, so a re-run resyncs - // cleanly. + // Re-entry-safe: a retry resumes from the persisted safe head. A + // transient sync failure leaves setup incomplete; the operator simply + // re-runs `setup`. // // Recovery rebuilds the batch tree from the checkpoint *after* this sync, so // the local tree is empty here. Disable frontier population for recovery's @@ -241,7 +260,11 @@ where input_reader .sync_to_current_safe_head() .await - .map_err(|e| RunError::Worker(WorkerExit::InputReader(InputReaderExit::Source(e))))?; + .map_err(|e| { + CommandError::Worker(WorkerExit::InputReader( + crate::commands::error::WorkerStop::Source(e), + )) + })?; // ── Branch: recovery rebuild vs the genesis-style detect-and-refuse ── let mut storage = storage::Storage::open(&db_path)?; @@ -263,12 +286,11 @@ where // ── Detection gate, steps 1–2: refuse if a previous instance left work ── // Read-only: no key, no L1 write. Runs *after* // the sync so step 2 reads a `safe_inputs` table populated to the safe - // head, and *before* the genesis snapshot / marker so a refusing setup - // leaves no marker — a re-run (while still incomplete) re-detects - // identically. (Once the marker *is* written, the idempotent early-return - // above skips the gate; that is correct — the deployment is this - // instance's own, and detecting a dirty chain is the job of a *fresh* - // `setup` whose marker is absent.) + // head, and *before* the genesis snapshot / completion transaction so + // a refusing setup cannot complete. A retry while still incomplete + // re-detects identically. Once setup is complete, + // plain `setup` is a no-op; detecting a dirty chain is the job of a + // fresh setup/rebuild. // // Coherence: read the submitter's `safe` nonce at the **persisted** safe // block from the sync, not the live `Safe` tag. The head can advance @@ -278,7 +300,7 @@ where // be missing from the not-yet-resynced scan. `pending` stays live, so any // submitter activity past the synced head still trips `pending > safe`. // - // F1: step 1 reads the LOCAL provider's pool view — a zombie tx dropped + // Step 1 reads the LOCAL provider's pool view — a zombie tx dropped // from this pool but alive elsewhere evades it, bounded at runtime by the // content-identity check (CanonicalDivergence → cockroach recovery). let synced_safe_block = storage.current_safe_block()?; @@ -297,28 +319,28 @@ where )?; // Refuse to register genesis over leftover recovery state. A - // `setup --recovery` that crashed before its marker leaves a non-zero + // `setup --recovery` that crashed before completion leaves a non-zero // batch-tree anchor (and maybe a root tip); booting genesis-style over // it would root the tree at the recovery nonce instead of 0. Fail loud - // (the marker is the only "this DB is mine" signal, and it's absent in - // both the fresh-genesis and crashed-recovery cases — so we check the - // anchor explicitly). Operator wipes the data dir and re-runs. + // Setup completion is absent in both the fresh-genesis and interrupted + // recovery cases, so we check the anchor explicitly. Operator wipes + // the data dir and re-runs. let anchor = storage.batch_tree_anchor()?; if anchor != 0 { return Err(SetupRecoveryError::GenesisOverRecoveryResidue { anchor }.into()); } // ── Genesis snapshot ───────────────────────────────────── - setup_fill::register_genesis_finalized_snapshot::( - genesis_app, - &mut storage, - &dumps_dir, - )?; + // Construct only after the admission facts and every + // detect-and-refuse gate. A panic leaves setup incomplete (the + // completion fact is never written, so the retry starts fresh), + // while completed no-ops and recovery never construct genesis + // state at all. + let genesis_app = genesis_app(); + fill::register_genesis_finalized_snapshot::(genesis_app, &mut storage, &dumps_dir)?; } - // ── Marker (the single linearization point for "setup finished") ── - storage.mark_setup_complete()?; - + // The caller commits setup_complete + Ready as one final transaction. tracing::info!( data_dir = %config.data_dir, chain_id = identity.chain_id, @@ -331,6 +353,62 @@ where Ok(()) } +enum SetupAdmission { + Proceed, + AlreadyComplete, +} + +fn admit_setup_lifecycle( + db_path: &str, + command: storage::LifecycleCommand, + recovery: bool, +) -> Result { + let populated = + std::path::Path::new(db_path).try_exists()? && !existing_sqlite_schema_is_empty(db_path)?; + if !populated { + storage::Storage::initialize_for_command(db_path, command)?; + return Ok(SetupAdmission::Proceed); + } + + // Admission facts only: divergence is absorbing; a completed setup + // is once-per-database (already-complete plain setup is a no-op, an + // already-complete rebuild is an error); a crashed prior attempt left + // only idempotent residue behind — the retry proceeds fresh over it. + let mut storage = storage::Storage::open_read_only(db_path)?; + if let Some((nonce, _)) = storage.canonical_divergence()? { + return Err(storage::LifecycleError::CanonicalDivergence { nonce }.into()); + } + if storage.is_setup_complete()? { + if recovery { + return Err(SetupRecoveryError::AlreadySetUp.into()); + } + return Ok(SetupAdmission::AlreadyComplete); + } + Ok(SetupAdmission::Proceed) +} + +fn settle_setup_lifecycle( + db_path: &str, + command: storage::LifecycleCommand, + result: &Result<(), CommandError>, +) -> Result<(), CommandError> { + match result { + Ok(()) => { + // The completion fact is part of the command, not telemetry: if + // it cannot be written, setup did not complete. + let mut storage = storage::Storage::open_writer(db_path)?; + storage.complete_setup()?; + Ok(()) + } + Err(_) => { + // Verdict-neutral black-box settlement: the recorder never + // replaces the command's own error. + crate::commands::record_terminal_fault_best_effort(db_path, command, result); + Ok(()) + } + } +} + /// The trusted checkpoint a `setup --recovery` folds from — the machine state /// `S` at block `B`, plus the two scalars the fold needs that the bare-metal app /// cannot recompute: `A` (`S`'s last-executed safe block — the fridge is the @@ -394,7 +472,7 @@ fn source_fold_inputs( checkpoint: &Checkpoint, stop_block: u64, submitter: Address, -) -> Result<(Vec, Vec), RunError> { +) -> Result<(Vec, Vec), CommandError> { let seeds = storage .safe_inputs_in_block_range(checkpoint.executed_safe_block, checkpoint.checkpoint_block)? .into_iter() @@ -431,7 +509,7 @@ async fn recover( input_reader: &mut InputReader, storage: &mut storage::Storage, dumps_dir: &std::path::Path, -) -> Result<(), RunError> +) -> Result<(), CommandError> where A: Application + 'static, { @@ -448,7 +526,8 @@ where let stop_block = flush_wallet_nonce(config, identity, timing.seconds_per_block, storage).await?; - // 3. Re-sync through C; F2 coherence. Frontier population stays OFF (the + // 3. Re-sync through C, then verify the resynced safe block reaches the + // flush observation before cascade. Frontier population stays OFF (the // caller disabled it before the initial sync): the tree is rebuilt in // step 6, so a frontier built here against an empty tree would falsely // diverge. `run`'s first sync populates it correctly once anchor = N'. @@ -456,8 +535,12 @@ where input_reader .sync_to_current_safe_head() .await - .map_err(|e| RunError::Worker(WorkerExit::InputReader(InputReaderExit::Source(e))))?; - let resynced_safe_block = storage.current_safe_block()?.unwrap_or(0); + .map_err(|e| { + CommandError::Worker(WorkerExit::InputReader( + crate::commands::error::WorkerStop::Source(e), + )) + })?; + let resynced_safe_block = require_resynced_safe_block(storage.current_safe_block()?)?; assert_resync_caught_up(resynced_safe_block, stop_block)?; // 4. Source the (A, B] direct seeds + the (B, C] replay stream. @@ -481,13 +564,13 @@ where seeds, replay, stop_block, - ); + )?; // 6. Fill the DB: finalized S', tree anchored at N', cursor past the ≤C // directs (already in S'). run boots from this state, and its first sync // populates the gold frontier from L1 with the anchor = N' (so the folded // `< N'` batches are skipped as trusted collapsed history, not foreign). - setup_fill::fill_recovery_state(recovered_app, resume_nonce, stop_block, storage, dumps_dir)?; + fill::fill_recovery_state(recovered_app, resume_nonce, stop_block, storage, dumps_dir)?; tracing::info!( executed_safe_block, @@ -499,6 +582,10 @@ where Ok(()) } +fn require_resynced_safe_block(observed: Option) -> Result { + observed.ok_or(SetupRecoveryError::MissingResyncedSafeHead) +} + /// Recovery step 2: flush the previous instance's stranded batch txs and wait /// for the wallet nonce to settle, returning `C` — the post-flush safe head at /// which every prior batch is resolved at safe depth. This is `setup`'s only @@ -513,7 +600,7 @@ async fn flush_wallet_nonce( identity: &DeploymentIdentity, seconds_per_block: u64, storage: &mut storage::Storage, -) -> Result { +) -> Result { // The recovery key must match the pinned batch-submitter address — flushing // under a different account's key would settle the wrong wallet's nonce (and // burn its gas) while recovery never settles `identity.batch_submitter_address`. @@ -525,23 +612,12 @@ async fn flush_wallet_nonce( // guarded constructor the runtime flush and `flush-mempool` use. let provider = crate::l1::provider::create_verified_signer_provider( &config.eth_rpc_url, - &key, + key.expose_secret(), config.chain_id, config.allow_insecure_rpc, ) .await - .map_err(|e| match e { - VerifiedSignerProviderError::ChainIdMismatch { rpc, expected } => { - RunError::Bootstrap(BootstrapError::ChainIdMismatch { - rpc, - config: expected, - }) - } - VerifiedSignerProviderError::ChainIdRpc(message) => { - RunError::Bootstrap(BootstrapError::ChainIdRpc { message }) - } - VerifiedSignerProviderError::Create(msg) => RunError::Io(std::io::Error::other(msg)), - })?; + .map_err(CommandError::from)?; let watermark = storage.wallet_nonce_watermark()?; let stop_block = MempoolFlusher::flush_to_safe( provider, @@ -578,7 +654,7 @@ fn run_detection_gate( batch_submitter: Address, checkpoint_block: u64, (pending_nonce, safe_nonce): (u64, u64), -) -> Result<(), RunError> { +) -> Result<(), CommandError> { // Step 1 — `pending > safe` means a previous instance left in-flight // (pending or mined-but-unsafe) batch txs. if pending_nonce > safe_nonce { @@ -592,8 +668,8 @@ fn run_detection_gate( // `safe_inputs` already reflect every previous batch (scanning them is // equivalent to scanning `(B, safe]`). Refuse if any batch-submitter tx // landed strictly past the checkpoint block. The query reads the - // reader-synced table, inheriting the reader's range-completeness hardening - // (F5) — see `first_batch_submitter_input_after_block`. + // reader-synced table, inheriting the reader's range-completeness + // hardening — see `first_batch_submitter_input_after_block`. if let Some((safe_input_index, found_block)) = storage.first_batch_submitter_input_after_block(batch_submitter, checkpoint_block)? { @@ -656,6 +732,26 @@ async fn read_submitter_nonce_views( Ok((pending, safe)) } +/// A first `Connection::open` can create the DB file before the transactional +/// baseline migration begins. That exact empty-schema crash state contains no +/// possible DB verdict and may resume setup. Any object at all makes the DB +/// non-empty; malformed/populated databases then take the strict read-only +/// lifecycle gate and are never migrated speculatively. +fn existing_sqlite_schema_is_empty( + db_path: &str, +) -> Result { + if std::fs::metadata(db_path)?.len() == 0 { + return Ok(true); + } + let connection = rusqlite::Connection::open_with_flags( + db_path, + rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY | rusqlite::OpenFlags::SQLITE_OPEN_NO_MUTEX, + )?; + let object_count: i64 = + connection.query_row("SELECT count(*) FROM sqlite_master", [], |row| row.get(0))?; + Ok(object_count == 0) +} + #[cfg(test)] mod tests { use super::*; @@ -663,6 +759,47 @@ mod tests { use crate::storage::StoredSafeInput; use crate::storage::test_helpers::{SENDER_A, default_protocol_timing, temp_db}; + #[test] + fn completed_plain_setup_is_a_noop_and_writes_nothing() { + let db = temp_db("setup-noop-preserves-recovery"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), storage::LifecycleCommand::Setup) + .expect("initialize"); + storage + .insert_initial_finalized_dump(&db._dir.path().join("finalized"), 0, 0, 0, 0) + .expect("register finalized snapshot"); + storage.complete_setup().expect("complete setup"); + storage + .record_terminal_fault(storage::LifecycleCommand::Run, "prior terminal death") + .expect("record prior fault"); + let before = storage + .latest_terminal_fault() + .expect("read") + .expect("recorded fault"); + drop(storage); + + assert!(matches!( + admit_setup_lifecycle(db.path.as_str(), storage::LifecycleCommand::Setup, false) + .expect("plain setup no-op"), + SetupAdmission::AlreadyComplete + )); + let after = Storage::open_read_only(db.path.as_str()) + .expect("reopen") + .latest_terminal_fault() + .expect("read") + .expect("still recorded"); + assert_eq!(after, before, "plain setup must write nothing"); + } + + #[test] + fn recovery_resync_requires_a_persisted_safe_head() { + assert_eq!(require_resynced_safe_block(Some(7)).unwrap(), 7); + assert!(matches!( + require_resynced_safe_block(None), + Err(SetupRecoveryError::MissingResyncedSafeHead) + )); + } + /// Seed `safe_inputs` with one batch-submitter (`SENDER_A`) input per block /// in `blocks`, synced up to the max block. Payloads are junk (scheduler /// no-ops) — detection scans raw rows, not the accepted frontier. @@ -697,7 +834,7 @@ mod tests { // Step 1 gates step 2: `pending > safe` refuses regardless of inputs // (and the DB is empty here, proving step 2 never ran). match run_detection_gate(&mut storage, SENDER_A, 0, (14, 13)) { - Err(RunError::Bootstrap(BootstrapError::SetupRefuse( + Err(CommandError::Bootstrap(BootstrapError::SetupRefuse( SetupRefuse::WalletNonceUnsettled { pending, safe }, ))) => assert_eq!((pending, safe), (14, 13)), other => panic!("expected WalletNonceUnsettled, got {other:?}"), @@ -710,7 +847,7 @@ mod tests { let mut storage = Storage::open(db.path.as_str()).expect("open"); seed_submitter_inputs(&mut storage, &[18]); match run_detection_gate(&mut storage, SENDER_A, 0, (1, 1)) { - Err(RunError::Bootstrap(BootstrapError::SetupRefuse( + Err(CommandError::Bootstrap(BootstrapError::SetupRefuse( SetupRefuse::BatchPastCheckpoint { checkpoint_block, found_block, @@ -794,7 +931,7 @@ mod tests { let result = flush_wallet_nonce(&config, &identity, 12, &mut storage).await; match result { - Err(RunError::Bootstrap(BootstrapError::Identity(IdentityError::Mismatch { + Err(CommandError::Bootstrap(BootstrapError::Identity(IdentityError::Mismatch { fields, .. }))) => assert_eq!(fields, "batch_submitter_address"), diff --git a/sequencer/src/runtime/test_support.rs b/sequencer/src/commands/test_support.rs similarity index 62% rename from sequencer/src/runtime/test_support.rs rename to sequencer/src/commands/test_support.rs index f12a0865..321cc0d2 100644 --- a/sequencer/src/runtime/test_support.rs +++ b/sequencer/src/commands/test_support.rs @@ -2,14 +2,18 @@ // SPDX-License-Identifier: Apache-2.0 (see LICENSE) //! Shared `#[cfg(test)]` fixtures for the runtime modules: a minimal -//! [`Application`] stub plus a dump-layout helper, used by both the worker -//! lifecycle tests ([`super::workers`]) and the setup-fill tests -//! ([`super::setup_fill`]). +//! [`Application`] stub plus a dump-layout helper, used by the worker +//! lifecycle tests (`run::workers`), the startup-hygiene tests +//! (`run::startup_hygiene`), and the setup-fill tests (`setup::fill`). use std::path::Path; use crate::ingress::inclusion_lane::dump_info::{self, create_dump_dir_with_info}; -use sequencer_core::application::{AppError, AppOutputs, Application, InvalidReason}; +use sequencer_core::application::{ + AppError, AppOutputs, Application, ApplicationProgress, ApplyInputCapability, InvalidReason, + ProgressCommitCapability, +}; +use sequencer_core::history::ExecutedInputCount; use sequencer_core::l2_tx::ValidUserOp; use sequencer_core::user_op::UserOp; @@ -17,7 +21,17 @@ use sequencer_core::user_op::UserOp; /// a directory with a marker file inside, `delete_dump` is /// `remove_dir_all`. The actual marker content is irrelevant — /// we only care about which directories exist post-sweep. -pub(crate) struct SweepTestApp; +#[derive(Clone, Default)] +pub(crate) struct SweepTestApp { + progress: ApplicationProgress, +} + +// Preserve the unit-struct-like fixture spelling used across runtime tests +// while carrying the scheduler-owned progress required by `Application`. +#[allow(non_upper_case_globals)] +pub(crate) const SweepTestApp: SweepTestApp = SweepTestApp { + progress: ApplicationProgress::new(ExecutedInputCount::ZERO, 0), +}; impl Application for SweepTestApp { const MAX_METHOD_PAYLOAD_BYTES: usize = 0; @@ -29,28 +43,33 @@ impl Application for SweepTestApp { ) -> Result<(), InvalidReason> { Ok(()) } - fn execute_valid_user_op( + fn apply_valid_user_op( &mut self, + _capability: ApplyInputCapability<'_>, _user_op: &ValidUserOp, _safe_block: u64, ) -> Result { Ok(Vec::new()) } - fn execute_direct_input( + fn apply_direct_input( &mut self, + _capability: ApplyInputCapability<'_>, _input: &sequencer_core::l2_tx::DirectInput, ) -> Result { unimplemented!("not used in these tests") } - fn executed_input_count(&self) -> u64 { - 0 + fn execution_progress(&self) -> &ApplicationProgress { + &self.progress } - fn last_executed_safe_block(&self) -> u64 { - 0 + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + &mut self.progress } fn from_dump(_prefix: &Path) -> Result { - Ok(SweepTestApp) + Ok(Self::default()) } fn create_dump(&self, prefix: &Path) -> Result<(), AppError> { std::fs::create_dir(prefix)?; @@ -70,7 +89,7 @@ impl Application for SweepTestApp { /// dir with `info.toml` + the stub app's dump under `state`. pub(crate) fn create_structured_dump(dump_dir: &std::path::Path) { create_dump_dir_with_info( - &SweepTestApp, + &SweepTestApp::default(), dump_dir, &dump_info::DumpInfo { format_version: dump_info::FORMAT_VERSION, diff --git a/sequencer/src/egress/api/health.rs b/sequencer/src/egress/api/health.rs index ac783c73..36462613 100644 --- a/sequencer/src/egress/api/health.rs +++ b/sequencer/src/egress/api/health.rs @@ -21,7 +21,7 @@ use serde::Serialize; use tokio::sync::mpsc; use crate::ingress::inclusion_lane::PendingUserOp; -use crate::runtime::shutdown::ShutdownSignal; +use crate::runtime::shutdown::RuntimeScope; /// Narrow health-check state. Holds only the signals the probes inspect; the /// `tx_sender` is a clone of the inclusion-lane channel and is closed iff the @@ -29,7 +29,7 @@ use crate::runtime::shutdown::ShutdownSignal; #[derive(Clone)] pub(crate) struct HealthState { pub tx_sender: mpsc::Sender, - pub shutdown: ShutdownSignal, + pub shutdown: RuntimeScope, } #[derive(Serialize)] @@ -76,7 +76,7 @@ mod tests { let (tx_sender, rx) = mpsc::channel::(1); let state = Arc::new(HealthState { tx_sender, - shutdown: ShutdownSignal::default(), + shutdown: RuntimeScope::default(), }); (state, rx) } @@ -99,6 +99,15 @@ mod tests { assert_eq!(readyz(State(state)).await, StatusCode::SERVICE_UNAVAILABLE); } + #[tokio::test] + async fn readyz_is_unavailable_after_storage_invariant_failure() { + let (state, _rx) = fresh_state(); + state + .shutdown + .contain_storage_invariant_failure("test fault"); + assert_eq!(readyz(State(state)).await, StatusCode::SERVICE_UNAVAILABLE); + } + #[tokio::test] async fn readyz_is_unavailable_when_lane_dropped() { let (state, rx) = fresh_state(); diff --git a/sequencer/src/egress/api/mod.rs b/sequencer/src/egress/api/mod.rs index 223fe801..39a7db9c 100644 --- a/sequencer/src/egress/api/mod.rs +++ b/sequencer/src/egress/api/mod.rs @@ -14,6 +14,9 @@ use std::sync::Arc; use axum::Router; use axum::routing::get; +use crate::runtime::shutdown::RuntimeScope; +use crate::storage::ReleaseScheduler; + pub(crate) use health::HealthState; pub use snapshot::SnapshotState; pub(crate) use state::SubscribeState; @@ -24,7 +27,9 @@ pub(crate) use state::SubscribeState; pub(crate) fn router( subscribe_state: Arc, health_state: Arc, - snapshot_state: Arc, + snapshot_state: SnapshotState, + shutdown: RuntimeScope, + snapshot_release_scheduler: ReleaseScheduler, ) -> Router { let subscribe_router = Router::new() .route("/ws/subscribe", get(subscribe::subscribe_l2_txs)) @@ -38,5 +43,9 @@ pub(crate) fn router( subscribe_router .merge(health_router) - .merge(snapshot::router(snapshot_state)) + .merge(snapshot::router( + snapshot_state, + shutdown, + snapshot_release_scheduler, + )) } diff --git a/sequencer/src/egress/api/snapshot.rs b/sequencer/src/egress/api/snapshot.rs index 0e6ad91d..84710608 100644 --- a/sequencer/src/egress/api/snapshot.rs +++ b/sequencer/src/egress/api/snapshot.rs @@ -38,7 +38,8 @@ use tokio::fs::File; use tokio::io::{AsyncRead, ReadBuf}; use tokio_util::io::ReaderStream; -use crate::storage::{LeaseGuard, LeasedDump, Storage}; +use crate::runtime::shutdown::RuntimeScope; +use crate::storage::{LeaseGuard, LeasedDump, ReleaseScheduler, Storage}; type BoxError = Box; @@ -52,7 +53,35 @@ pub struct SnapshotState { pub state_file_in_dump: fn(&Path) -> PathBuf, } -pub(crate) fn router(state: Arc) -> Router { +struct SnapshotApiState { + snapshot: SnapshotState, + shutdown: RuntimeScope, + release_scheduler: ReleaseScheduler, +} + +impl SnapshotApiState { + /// Refuse to start a stream only after a terminal fault is contained. + /// Streaming an already-immutable operator snapshot is not + /// authority-bearing (ADR), so ordinary graceful shutdown does NOT gate + /// these routes — the watchdog's byte-compare poll and indexer fetches + /// keep working through an operator drain. Containment is checked + /// at stream start; the state a contained fault may have poisoned must + /// not be served. + fn authorize_stream(&self) -> Option> { + self.shutdown.authorize() + } +} + +pub(crate) fn router( + snapshot: SnapshotState, + shutdown: RuntimeScope, + release_scheduler: ReleaseScheduler, +) -> Router { + let state = Arc::new(SnapshotApiState { + snapshot, + shutdown, + release_scheduler, + }); Router::new() .route("/finalized_state", get(finalized_state)) .route( @@ -71,43 +100,57 @@ struct InclusionBlockResponse { /// `GET /finalized_state/inclusion_block` — cheap read, no lease (no file is /// opened). 404 if no finalized snapshot exists. -async fn finalized_inclusion_block(State(state): State>) -> Response { - let db_path = state.db_path.clone(); - let result = tokio::task::spawn_blocking(move || -> Result<_, BoxError> { +async fn finalized_inclusion_block(State(state): State>) -> Response { + let Some(_auth) = state.authorize_stream() else { + return StatusCode::SERVICE_UNAVAILABLE.into_response(); + }; + let db_path = state.snapshot.db_path.clone(); + let result = storage_task(&state, "read finalized inclusion block", move |_scope| { Ok(Storage::open_read_only(&db_path)?.finalized_dump()?) }) .await; match result { - Ok(Ok(Some(finalized))) => Json(InclusionBlockResponse { + Ok(Some(finalized)) => Json(InclusionBlockResponse { inclusion_block: finalized.inclusion_block, l2_tx_index: finalized.l2_tx_index, }) .into_response(), - Ok(Ok(None)) => StatusCode::NOT_FOUND.into_response(), - Ok(Err(err)) => internal_error("read finalized inclusion block", err), - Err(join) => internal_error("inclusion-block task join", join), + Ok(None) => StatusCode::NOT_FOUND.into_response(), + Err(err) => internal_error("read finalized inclusion block", err), } } /// `GET /finalized_state` — stream the finalized state file (watchdog /// source). Supports `If-None-Match` against `"block-"` for a 304. -async fn finalized_state(State(state): State>, headers: HeaderMap) -> Response { +async fn finalized_state( + State(state): State>, + headers: HeaderMap, +) -> Response { + let Some(_auth) = state.authorize_stream() else { + return StatusCode::SERVICE_UNAVAILABLE.into_response(); + }; let leased = match acquire_finalized(&state).await { Ok(Some(leased)) => leased, Ok(None) => return StatusCode::NOT_FOUND.into_response(), Err(err) => return internal_error("acquire finalized lease", err), }; - let inclusion_block = leased - .inclusion_block - .expect("finalized snapshot carries an inclusion block"); + let Some(inclusion_block) = leased.inclusion_block else { + state + .shutdown + .contain_storage_invariant_failure("finalized snapshot carried no inclusion block"); + return internal_error( + "read finalized snapshot", + "finalized snapshot carried no inclusion block", + ); + }; let etag = format!("\"block-{inclusion_block}\""); if if_none_match(&headers, &etag) { // 304: dropping `leased` here releases the lease via its guard. return StatusCode::NOT_MODIFIED.into_response(); } - let path = (state.state_file_in_dump)(&leased.prefix); + let path = (state.snapshot.state_file_in_dump)(&leased.prefix); let l2_tx_index = leased.l2_tx_index; let LeasedDump { guard, .. } = leased; @@ -121,20 +164,31 @@ async fn finalized_state(State(state): State>, headers: Heade .body(stream_body(file, guard)) .expect("snapshot response headers are well-formed"), // `guard` is a local here; on this error path it drops → lease released. - Err(err) => internal_error("open finalized state file", err), + Err(err) => { + if err.kind() == std::io::ErrorKind::NotFound { + tracing::error!(path = ?path, "durable finalized snapshot artifact is missing"); + state.shutdown.contain_storage_invariant_failure(format!( + "durable finalized snapshot artifact missing: {path:?}" + )); + } + internal_error("open finalized state file", err) + } } } /// `GET /latest_snapshot` — stream the latest snapshot dump (indexers: fetch /// then subscribe at this offset). Latest pending if any, else finalized. -async fn latest_snapshot(State(state): State>) -> Response { +async fn latest_snapshot(State(state): State>) -> Response { + let Some(_auth) = state.authorize_stream() else { + return StatusCode::SERVICE_UNAVAILABLE.into_response(); + }; let leased = match acquire_latest(&state).await { Ok(Some(leased)) => leased, Ok(None) => return StatusCode::NOT_FOUND.into_response(), Err(err) => return internal_error("acquire latest snapshot lease", err), }; - let path = (state.state_file_in_dump)(&leased.prefix); + let path = (state.snapshot.state_file_in_dump)(&leased.prefix); let l2_tx_index = leased.l2_tx_index; let LeasedDump { guard, .. } = leased; @@ -145,7 +199,15 @@ async fn latest_snapshot(State(state): State>) -> Response { .header("X-L2-Tx-Index", l2_tx_index.to_string()) .body(stream_body(file, guard)) .expect("snapshot response headers are well-formed"), - Err(err) => internal_error("open latest snapshot file", err), + Err(err) => { + if err.kind() == std::io::ErrorKind::NotFound { + tracing::error!(path = ?path, "durable latest snapshot artifact is missing"); + state.shutdown.contain_storage_invariant_failure(format!( + "durable latest snapshot artifact missing: {path:?}" + )); + } + internal_error("open latest snapshot file", err) + } } } @@ -156,40 +218,81 @@ fn stream_body(file: File, guard: LeaseGuard) -> Body { })) } -// ── Lease acquisition (storage returns the dump bundled with its release) ── +// ── Blocking storage tasks ───────────────────────────────────────────────── -async fn acquire_finalized(state: &SnapshotState) -> Result, BoxError> { - let db_path = state.db_path.clone(); - tokio::task::spawn_blocking(move || -> Result, BoxError> { - let mut storage = Storage::open_writer(&db_path)?; - Ok(storage.acquire_finalized_lease(spawn_blocking_release)?) +/// One spawn/join/classify shape for this endpoint's blocking storage work. +/// The posture is deliberate and stays local: an HTTP handler has no +/// worker-exit channel to carry a typed error to the supervisor, so a +/// persistent row/schema failure or a storage-task panic contains +/// immediately. +async fn storage_task( + state: &SnapshotApiState, + operation: &'static str, + work: F, +) -> Result +where + T: Send + 'static, + F: FnOnce(crate::runtime::shutdown::RuntimeScope) -> Result + Send + 'static, +{ + let scope = state.shutdown.clone(); + match tokio::task::spawn_blocking(move || { + // Independent retention, bound first so it drops last: the task owns + // data-directory exclusivity for its REAL lifetime — including the + // final drop of its SQLite connection (a WAL checkpoint writes to + // the data dir) — regardless of what `work` does with its scope + // argument. The lease closures consume theirs early (the reporter + // Arc can die inside storage on the None/Err paths), which is + // exactly the coupling this binding exists to break (ADR §1; found + // by adversarial review). + let _runtime_lifetime = scope.clone(); + work(scope) }) - .await? + .await + { + Ok(Ok(value)) => Ok(value), + Ok(Err(error)) => { + if persistent_storage_error(error.as_ref(), operation) { + state + .shutdown + .contain_storage_invariant_failure(format!("{operation}: {error}")); + } + Err(error) + } + Err(join) => { + if storage_task_panicked(&join, operation) { + state.shutdown.contain_storage_invariant_failure(format!( + "{operation}: task panicked: {join}" + )); + } + Err(Box::new(join)) + } + } } -async fn acquire_latest(state: &SnapshotState) -> Result, BoxError> { - let db_path = state.db_path.clone(); - tokio::task::spawn_blocking(move || -> Result, BoxError> { +// ── Lease acquisition (storage returns the dump bundled with its release) ── + +async fn acquire_finalized(state: &SnapshotApiState) -> Result, BoxError> { + let db_path = state.snapshot.db_path.clone(); + let release_scheduler = state.release_scheduler.clone(); + storage_task(state, "acquire finalized snapshot lease", move |scope| { + let report_persistent_failure: crate::storage::PersistentReleaseFailureReporter = + Arc::new(move |cause: &str| scope.contain_storage_invariant_failure(cause)); let mut storage = Storage::open_writer(&db_path)?; - Ok(storage.acquire_latest_snapshot_lease(spawn_blocking_release)?) + Ok(storage.acquire_finalized_lease(release_scheduler, report_persistent_failure)?) }) - .await? + .await } -// ── Release scheduler: keep the lease release off the async worker ───────── - -/// The `ReleaseScheduler` the egress layer hands to `acquire_*_lease`. The -/// lease release is a write-lock-contended SQLite write, and on client -/// disconnect the guard drops on an async worker thread — so offload it to the -/// blocking pool rather than stall the worker. Off-runtime (shouldn't happen -/// from here), run it inline. -fn spawn_blocking_release(release: Box) { - match tokio::runtime::Handle::try_current() { - Ok(handle) => { - handle.spawn_blocking(release); - } - Err(_) => release(), - } +async fn acquire_latest(state: &SnapshotApiState) -> Result, BoxError> { + let db_path = state.snapshot.db_path.clone(); + let release_scheduler = state.release_scheduler.clone(); + storage_task(state, "acquire latest snapshot lease", move |scope| { + let report_persistent_failure: crate::storage::PersistentReleaseFailureReporter = + Arc::new(move |cause: &str| scope.contain_storage_invariant_failure(cause)); + let mut storage = Storage::open_writer(&db_path)?; + Ok(storage.acquire_latest_snapshot_lease(release_scheduler, report_persistent_failure)?) + }) + .await } // ── Streaming body that owns the lease guard ─────────────────────────────── @@ -227,3 +330,185 @@ fn internal_error(context: &str, err: impl std::fmt::Display) -> Response { tracing::warn!(error = %err, context, "snapshot endpoint failed"); StatusCode::INTERNAL_SERVER_ERROR.into_response() } + +fn storage_task_panicked(join: &tokio::task::JoinError, operation: &'static str) -> bool { + if join.is_panic() { + tracing::error!(operation, "persistent storage invariant violation"); + true + } else { + false + } +} + +fn persistent_storage_error( + mut error: &(dyn std::error::Error + 'static), + operation: &'static str, +) -> bool { + loop { + let persistent = error + .downcast_ref::() + .is_some_and(crate::storage::is_persistent_storage_error) + || error + .downcast_ref::() + .is_some_and(crate::storage::is_persistent_storage_open_error); + if persistent { + tracing::error!(operation, error = %error, "persistent storage invariant violation"); + return true; + } + let Some(source) = error.source() else { + return false; + }; + error = source; + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::storage::test_helpers::temp_db; + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn corrupt_finalized_snapshot_trips_terminal_storage_fault() { + let db = temp_db("corrupt-finalized-endpoint"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + storage + .insert_finalized_dump(Path::new("/tmp/corrupt-finalized"), 12, 34) + .expect("insert finalized snapshot"); + drop(storage); + + let conn = Storage::open_connection(db.path.as_str()).expect("raw connection"); + conn.pragma_update(None, "ignore_check_constraints", "ON") + .expect("allow corruption fixture"); + conn.execute( + "UPDATE finalized_snapshot SET l2_tx_index = -1 WHERE singleton_id = 0", + [], + ) + .expect("corrupt finalized cursor"); + drop(conn); + + let shutdown = RuntimeScope::default(); + let state = Arc::new(SnapshotApiState { + snapshot: SnapshotState { + db_path: db.path, + state_file_in_dump: |prefix| prefix.join("state"), + }, + shutdown: shutdown.clone(), + release_scheduler: Arc::new(|release| release()), + }); + + let response = finalized_inclusion_block(State(state)).await; + + assert_eq!(response.status(), StatusCode::INTERNAL_SERVER_ERROR); + assert!(shutdown.is_storage_invariant_contained()); + assert!(shutdown.is_shutdown_requested()); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn graceful_shutdown_does_not_gate_snapshot_reads_but_containment_does() { + let db = temp_db("snapshot-gate-predicate"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + storage + .insert_finalized_dump(Path::new("/tmp/gate-finalized"), 12, 34) + .expect("insert finalized snapshot"); + drop(storage); + + let shutdown = RuntimeScope::default(); + let state = Arc::new(SnapshotApiState { + snapshot: SnapshotState { + db_path: db.path, + state_file_in_dump: |prefix| prefix.join("state"), + }, + shutdown: shutdown.clone(), + release_scheduler: Arc::new(|release| release()), + }); + + // Immutable operator reads are not authority-bearing (ADR): an + // ordinary graceful drain keeps serving the watchdog's poll. + shutdown.request_shutdown(); + let response = finalized_inclusion_block(State(state.clone())).await; + assert_eq!(response.status(), StatusCode::OK); + + // A contained terminal fault is the one condition that refuses a + // stream start: the state it may have poisoned must not be served. + shutdown.contain_storage_invariant_failure("test containment"); + let response = finalized_inclusion_block(State(state)).await; + assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn dangling_finalized_snapshot_row_trips_terminal_storage_fault() { + let db = temp_db("dangling-finalized-endpoint"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + storage + .insert_finalized_dump(Path::new("/tmp/dangling-finalized"), 12, 34) + .expect("insert finalized snapshot"); + drop(storage); + + let conn = Storage::open_connection(db.path.as_str()).expect("raw connection"); + conn.pragma_update(None, "foreign_keys", "OFF") + .expect("disable foreign keys for corruption fixture"); + conn.execute("DELETE FROM dumps", []) + .expect("remove referenced dump row"); + drop(conn); + + let shutdown = RuntimeScope::default(); + let state = Arc::new(SnapshotApiState { + snapshot: SnapshotState { + db_path: db.path, + state_file_in_dump: |prefix| prefix.join("state"), + }, + shutdown: shutdown.clone(), + release_scheduler: Arc::new(|release| release()), + }); + + let response = finalized_inclusion_block(State(state)).await; + assert_eq!(response.status(), StatusCode::INTERNAL_SERVER_ERROR); + assert!(shutdown.is_storage_invariant_contained()); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn missing_finalized_snapshot_file_trips_terminal_storage_fault() { + let db = temp_db("missing-finalized-file"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let missing_root = tempfile::tempdir().expect("missing snapshot parent"); + let missing_prefix = missing_root.path().join("not-created"); + storage + .insert_finalized_dump(&missing_prefix, 12, 34) + .expect("insert finalized snapshot"); + drop(storage); + + let shutdown = RuntimeScope::default(); + let state = Arc::new(SnapshotApiState { + snapshot: SnapshotState { + db_path: db.path, + state_file_in_dump: |prefix| prefix.join("state"), + }, + shutdown: shutdown.clone(), + release_scheduler: Arc::new(|release| release()), + }); + + let response = finalized_state(State(state), HeaderMap::new()).await; + assert_eq!(response.status(), StatusCode::INTERNAL_SERVER_ERROR); + assert!(shutdown.is_storage_invariant_contained()); + } + + #[tokio::test] + async fn transient_storage_open_error_does_not_trip_terminal_fault() { + let shutdown = RuntimeScope::default(); + let error = crate::storage::StorageOpenError::Sqlite(rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code: rusqlite::ffi::ErrorCode::DatabaseBusy, + extended_code: 5, + }, + None, + )); + + assert!(!persistent_storage_error( + &error, + "test transient storage contention" + )); + + assert!(!shutdown.is_storage_invariant_contained()); + assert!(!shutdown.is_shutdown_requested()); + } +} diff --git a/sequencer/src/egress/api/state.rs b/sequencer/src/egress/api/state.rs index 1edb5c11..078cd6fa 100644 --- a/sequencer/src/egress/api/state.rs +++ b/sequencer/src/egress/api/state.rs @@ -9,11 +9,11 @@ use tokio::sync::{OwnedSemaphorePermit, Semaphore}; use crate::egress::l2_tx_feed::L2TxFeed; use crate::http::ApiError; -use crate::runtime::shutdown::ShutdownSignal; +use crate::runtime::shutdown::RuntimeScope; #[derive(Clone)] pub(crate) struct SubscribeState { - pub shutdown: ShutdownSignal, + pub shutdown: RuntimeScope, pub ws_subscriber_limit: Arc, pub ws_max_catchup_events: u64, pub tx_feed: L2TxFeed, @@ -21,7 +21,7 @@ pub(crate) struct SubscribeState { impl SubscribeState { pub(crate) fn new( - shutdown: ShutdownSignal, + shutdown: RuntimeScope, tx_feed: L2TxFeed, ws_max_subscribers: usize, ws_max_catchup_events: u64, diff --git a/sequencer/src/egress/api/subscribe.rs b/sequencer/src/egress/api/subscribe.rs index ab2abb3c..aa93db68 100644 --- a/sequencer/src/egress/api/subscribe.rs +++ b/sequencer/src/egress/api/subscribe.rs @@ -59,7 +59,11 @@ async fn run_ws_session( _subscriber_permit: OwnedSemaphorePermit, ws_max_catchup_events: u64, ) { - let mut subscription = match tx_feed.subscribe_from(from_offset, ws_max_catchup_events) { + let shutdown = tx_feed.runtime_scope(); + let mut subscription = match tx_feed + .subscribe_from(from_offset, ws_max_catchup_events) + .await + { Ok(subscription) => subscription, Err(SubscribeError::CatchUpWindowExceeded { requested_offset, @@ -75,28 +79,53 @@ async fn run_ws_session( let reason = format!( "{WS_CATCHUP_WINDOW_EXCEEDED_REASON}: live_start_offset={live_start_offset}" ); - close_with_frame(&mut socket, close_code::POLICY, reason.as_str()).await; + close_with_frame(&mut socket, close_code::POLICY, reason.as_str(), &shutdown).await; return; } Err(SubscribeError::OpenStorage { source }) => { warn!(error = %source, "ws subscription failed to open replay storage"); - close_with_frame(&mut socket, close_code::ERROR, "subscription unavailable").await; + close_with_frame( + &mut socket, + close_code::ERROR, + "subscription unavailable", + &shutdown, + ) + .await; return; } Err(SubscribeError::LoadHeadOffset { source }) => { warn!(error = %source, "ws subscription failed to read replay head"); - close_with_frame(&mut socket, close_code::ERROR, "subscription unavailable").await; + close_with_frame( + &mut socket, + close_code::ERROR, + "subscription unavailable", + &shutdown, + ) + .await; + return; + } + Err(SubscribeError::StorageInvariantViolation) => { + warn!("ws subscription encountered a persistent storage invariant failure"); + close_with_frame( + &mut socket, + close_code::ERROR, + "subscription unavailable", + &shutdown, + ) + .await; return; } }; loop { tokio::select! { + biased; + _ = shutdown.wait_for_shutdown() => break, maybe_event = subscription.recv() => { let Some(event) = maybe_event else { break; }; - if send_ws_event(&mut socket, &event).await.is_err() { + if send_ws_event(&mut socket, &event, &shutdown).await.is_err() { break; } } @@ -104,7 +133,10 @@ async fn run_ws_session( match inbound { Some(Ok(Message::Close(_))) | None => break, Some(Ok(Message::Ping(payload))) => { - if socket.send(Message::Pong(payload)).await.is_err() { + if send_ws_message(&mut socket, Message::Pong(payload), &shutdown) + .await + .is_err() + { break; } } @@ -120,16 +152,28 @@ async fn run_ws_session( } } -async fn close_with_frame(socket: &mut WebSocket, code: u16, reason: &str) { - let _ = socket - .send(Message::Close(Some(CloseFrame { +async fn close_with_frame( + socket: &mut WebSocket, + code: u16, + reason: &str, + shutdown: &crate::runtime::shutdown::RuntimeScope, +) { + let _ = send_ws_message( + socket, + Message::Close(Some(CloseFrame { code, reason: reason.into(), - }))) - .await; + })), + shutdown, + ) + .await; } -async fn send_ws_event(socket: &mut WebSocket, event: &BroadcastTxMessage) -> Result<(), ()> { +async fn send_ws_event( + socket: &mut WebSocket, + event: &BroadcastTxMessage, + shutdown: &crate::runtime::shutdown::RuntimeScope, +) -> Result<(), ()> { let payload = match serde_json::to_string(event) { Ok(value) => value, Err(err) => { @@ -138,8 +182,29 @@ async fn send_ws_event(socket: &mut WebSocket, event: &BroadcastTxMessage) -> Re } }; - if socket.send(Message::Text(payload.into())).await.is_err() { + send_ws_message(socket, Message::Text(payload.into()), shutdown).await +} + +/// The WS externalization primitive: emitting requires the token, so a new +/// frame-sending site cannot skip the containment consult. +async fn send_ws_message( + socket: &mut WebSocket, + message: Message, + shutdown: &crate::runtime::shutdown::RuntimeScope, +) -> Result<(), ()> { + let Some(auth) = shutdown.authorize() else { return Err(()); + }; + send_authorized(auth, socket, message).await +} + +async fn send_authorized( + _auth: crate::runtime::shutdown::Authorized<'_>, + socket: &mut WebSocket, + message: Message, +) -> Result<(), ()> { + match socket.send(message).await { + Ok(()) => Ok(()), + Err(_) => Err(()), } - Ok(()) } diff --git a/sequencer/src/egress/l2_tx_feed/error.rs b/sequencer/src/egress/l2_tx_feed/error.rs index 778dc39a..4f708e35 100644 --- a/sequencer/src/egress/l2_tx_feed/error.rs +++ b/sequencer/src/egress/l2_tx_feed/error.rs @@ -3,7 +3,9 @@ use thiserror::Error; -use crate::storage::StorageOpenError; +use crate::storage::{ + StorageOpenError, is_persistent_storage_error, is_persistent_storage_open_error, +}; #[derive(Debug, Error)] pub enum SubscribeError { @@ -17,6 +19,8 @@ pub enum SubscribeError { #[source] source: rusqlite::Error, }, + #[error("persistent storage invariant violation while preparing subscription")] + StorageInvariantViolation, #[error( "catch-up window exceeded: requested offset {requested_offset}, live start {live_start_offset}, max {max_catchup_events}" )] @@ -27,6 +31,17 @@ pub enum SubscribeError { }, } +impl SubscribeError { + pub(super) fn is_persistent_storage_invariant(&self) -> bool { + match self { + Self::OpenStorage { source } => open_error_is_persistent(source), + Self::LoadHeadOffset { source } => is_persistent_storage_error(source), + Self::StorageInvariantViolation => true, + Self::CatchUpWindowExceeded { .. } => false, + } + } +} + #[derive(Debug, Error)] pub enum SubscriptionError { #[error("cannot open subscription storage")] @@ -40,9 +55,26 @@ pub enum SubscriptionError { #[source] source: rusqlite::Error, }, + #[error("persistent storage invariant violation while reading subscription")] + StorageInvariantViolation, #[error("subscription task join error: {source}")] Join { #[source] source: tokio::task::JoinError, }, } + +impl SubscriptionError { + pub(super) fn is_persistent_storage_invariant(&self) -> bool { + match self { + Self::OpenStorage { source } => open_error_is_persistent(source), + Self::LoadReplay { source, .. } => is_persistent_storage_error(source), + Self::StorageInvariantViolation => true, + Self::Join { source } => source.is_panic(), + } + } +} + +fn open_error_is_persistent(error: &StorageOpenError) -> bool { + is_persistent_storage_open_error(error) +} diff --git a/sequencer/src/egress/l2_tx_feed/mod.rs b/sequencer/src/egress/l2_tx_feed/mod.rs index 6820b350..c2350e50 100644 --- a/sequencer/src/egress/l2_tx_feed/mod.rs +++ b/sequencer/src/egress/l2_tx_feed/mod.rs @@ -11,19 +11,33 @@ mod tests; pub use error::{SubscribeError, SubscriptionError}; pub use sequencer_core::broadcast::BroadcastTxMessage; +use std::panic::{AssertUnwindSafe, catch_unwind}; use std::time::Duration; use alloy_primitives::Address; use tokio::sync::mpsc; -use crate::runtime::shutdown::ShutdownSignal; +use crate::runtime::process_lock::spawn_blocking_with_lock; +use crate::runtime::shutdown::RuntimeScope; use crate::storage::{OrderedL2TxRow, Storage}; +/// Best-effort extraction of a panic payload's message for fault causes. +fn panic_message(payload: &dyn std::any::Any) -> &str { + payload + .downcast_ref::<&str>() + .copied() + .or_else(|| payload.downcast_ref::().map(String::as_str)) + .unwrap_or("non-string panic payload") +} + #[derive(Debug, Clone, Copy)] pub struct L2TxFeedConfig { pub idle_poll_interval: Duration, pub page_size: usize, - pub batch_submitter_address: Option
, + /// Address of the batch submitter wallet. Direct inputs from this sender + /// are skipped before WS delivery (they're our own batch submissions). + /// One of I11's three consumer-side sender checks — keep them in sync. + pub batch_submitter_address: Address, } #[derive(Clone)] @@ -31,14 +45,16 @@ pub struct L2TxFeed { db_path: String, page_size: usize, idle_poll_interval: Duration, - batch_submitter_address: Option
, - shutdown: ShutdownSignal, + batch_submitter_address: Address, + shutdown: RuntimeScope, } pub struct Subscription { receiver: mpsc::Receiver, task: Option, - shutdown: ShutdownSignal, + /// Pure notification half: the subscription only waits for stop. The + /// streaming task holds the scope (and with it the lock) itself. + shutdown: crate::runtime::shutdown::ShutdownSignal, } type SubscriptionTask = tokio::task::JoinHandle>; @@ -47,18 +63,20 @@ const DEFAULT_IDLE_POLL_INTERVAL: Duration = Duration::from_millis(20); const DEFAULT_PAGE_SIZE: usize = 256; const SUBSCRIPTION_BUFFER_CAPACITY: usize = 1024; -impl Default for L2TxFeedConfig { - fn default() -> Self { +impl L2TxFeedConfig { + /// The only constructor: the submitter address is mandatory, so a feed + /// that fans out our own batch envelopes is unconstructible. + pub fn new(batch_submitter_address: Address) -> Self { Self { idle_poll_interval: DEFAULT_IDLE_POLL_INTERVAL, page_size: DEFAULT_PAGE_SIZE, - batch_submitter_address: None, + batch_submitter_address, } } } impl L2TxFeed { - pub fn new(db_path: String, shutdown: ShutdownSignal, config: L2TxFeedConfig) -> Self { + pub fn new(db_path: String, shutdown: RuntimeScope, config: L2TxFeedConfig) -> Self { Self { db_path, page_size: config.page_size.max(1), @@ -68,17 +86,66 @@ impl L2TxFeed { } } - pub fn subscribe_from( + pub async fn subscribe_from( &self, from_offset: u64, max_catchup_events: u64, ) -> Result { - let (head_offset, catchup_events) = load_catchup_info( - self.db_path.as_str(), - from_offset, - max_catchup_events, - self.batch_submitter_address, - )?; + // Blocking SQLite (an open plus a COUNT over up to + // `max_catchup_events` rows) runs on the blocking pool, making this + // signature's `async` honest; the join classifies a decoder panic, so + // the prepare phase needs no inline `catch_unwind`. The + // streaming task below keeps its `catch_unwind` deliberately: its + // only join point is `Subscription::finish`, and containment must + // fire at the fault, not when the socket unwinds. Cancelling the + // awaiting WS task detaches started blocking work, so the prepare + // closure independently retains the process lock until its SQLite + // work ends. + let prepare = { + let db_path = self.db_path.clone(); + let batch_submitter_address = self.batch_submitter_address; + spawn_blocking_with_lock(self.shutdown.process_lock(), move || { + load_catchup_info( + db_path.as_str(), + from_offset, + max_catchup_events, + batch_submitter_address, + ) + }) + .await + }; + let (head_offset, catchup_events) = match prepare { + Ok(Ok(info)) => info, + Ok(Err(error)) if error.is_persistent_storage_invariant() => { + tracing::error!( + error = %error, + "persistent storage invariant violation while preparing tx-feed subscription" + ); + self.shutdown.contain_storage_invariant_failure(format!( + "preparing tx-feed subscription: {error}" + )); + return Err(SubscribeError::StorageInvariantViolation); + } + Ok(Err(error)) => return Err(error), + Err(join) if join.is_panic() => { + let payload = join.into_panic(); + let message = panic_message(&*payload); + tracing::error!( + panic = message, + "storage invariant violation while preparing tx-feed subscription" + ); + self.shutdown.contain_storage_invariant_failure(format!( + "panic preparing tx-feed subscription: {message}" + )); + return Err(SubscribeError::StorageInvariantViolation); + } + // Not a panic: the runtime is tearing down and cancelled the + // blocking task before it started. Nothing to contain. + Err(join) => { + tracing::warn!(error = %join, "tx-feed prepare task did not run"); + return Err(SubscribeError::StorageInvariantViolation); + } + }; if catchup_events > max_catchup_events { return Err(SubscribeError::CatchUpWindowExceeded { requested_offset: from_offset, @@ -94,28 +161,58 @@ impl L2TxFeed { let batch_submitter_address = self.batch_submitter_address; let shutdown = self.shutdown.clone(); let task = tokio::task::spawn_blocking(move || { - run_subscription( - db_path.as_str(), - page_size, - idle_poll_interval, - batch_submitter_address, - from_offset, - shutdown, - events_tx, - ) + match catch_unwind(AssertUnwindSafe(|| { + run_subscription( + db_path.as_str(), + page_size, + idle_poll_interval, + batch_submitter_address, + from_offset, + shutdown.clone(), + events_tx, + ) + })) { + Ok(Err(error)) if error.is_persistent_storage_invariant() => { + tracing::error!( + error = %error, + "persistent storage invariant violation while reading tx-feed subscription" + ); + shutdown.contain_storage_invariant_failure(format!( + "reading tx-feed subscription: {error}" + )); + Err(SubscriptionError::StorageInvariantViolation) + } + Ok(result) => result, + Err(payload) => { + let message = panic_message(&*payload); + tracing::error!( + panic = message, + "storage invariant violation while reading tx-feed subscription" + ); + shutdown.contain_storage_invariant_failure(format!( + "panic reading tx-feed subscription: {message}" + )); + Err(SubscriptionError::StorageInvariantViolation) + } + } }); Ok(Subscription { receiver: events_rx, task: Some(task), - shutdown: self.shutdown.clone(), + shutdown: self.shutdown.signal(), }) } + + pub(crate) fn runtime_scope(&self) -> RuntimeScope { + self.shutdown.clone() + } } impl Subscription { pub async fn recv(&mut self) -> Option { tokio::select! { + biased; _ = self.shutdown.wait_for_shutdown() => None, maybe_event = self.receiver.recv() => maybe_event, } @@ -145,7 +242,7 @@ fn load_catchup_info( db_path: &str, from_offset: u64, max_catchup_events: u64, - batch_submitter_address: Option
, + batch_submitter_address: Address, ) -> Result<(u64, u64), SubscribeError> { let mut storage = Storage::open_read_only(db_path) .map_err(|source| SubscribeError::OpenStorage { source })?; @@ -166,9 +263,9 @@ fn run_subscription( db_path: &str, page_size: usize, idle_poll_interval: Duration, - batch_submitter_address: Option
, + batch_submitter_address: Address, from_offset: u64, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, events_tx: mpsc::Sender, ) -> Result<(), SubscriptionError> { let mut storage = Storage::open_read_only(db_path) @@ -205,6 +302,7 @@ fn run_subscription( nonce, safe_block, batch_nonce, + .. } => BroadcastTxMessage::from_user_op(offset, tx, nonce, safe_block, batch_nonce), OrderedL2TxRow::DirectInput { offset, @@ -215,7 +313,7 @@ fn run_subscription( transaction_hash, .. } => { - if batch_submitter_address == Some(tx.sender) { + if tx.sender == batch_submitter_address { continue; } BroadcastTxMessage::from_direct_input( diff --git a/sequencer/src/egress/l2_tx_feed/tests.rs b/sequencer/src/egress/l2_tx_feed/tests.rs index e1e389e6..4494429e 100644 --- a/sequencer/src/egress/l2_tx_feed/tests.rs +++ b/sequencer/src/egress/l2_tx_feed/tests.rs @@ -6,9 +6,10 @@ use std::time::{Duration, SystemTime}; use alloy_primitives::{Address, B256, Signature}; use tokio::sync::oneshot; -use super::{BroadcastTxMessage, L2TxFeed, L2TxFeedConfig, SubscribeError}; +use super::{BroadcastTxMessage, L2TxFeed, L2TxFeedConfig, SubscribeError, SubscriptionError}; use crate::ingress::inclusion_lane::{PendingUserOp, SequencerError}; -use crate::runtime::shutdown::ShutdownSignal; +use crate::runtime::process_lock::{ProcessLock, ProcessLockError}; +use crate::runtime::shutdown::RuntimeScope; use crate::storage::test_helpers::temp_db; use crate::storage::{FrontierMode, IngestedSafeInput, SafeInputRange, Storage, StoredSafeInput}; use sequencer_core::l2_tx::{DirectInput, ValidUserOp}; @@ -68,9 +69,9 @@ async fn subscribe_from_rejects_catchup_window() { let db = temp_db("catchup-window"); seed_ordered_txs(db.path.as_str()); append_direct_input(db.path.as_str()); - let feed = test_feed(db.path.as_str(), ShutdownSignal::default()); + let feed = test_feed(db.path.as_str(), RuntimeScope::default()); - let result = feed.subscribe_from(1, 1); + let result = feed.subscribe_from(1, 1).await; assert!(matches!( result, @@ -86,9 +87,9 @@ async fn subscribe_from_rejects_catchup_window() { async fn subscribe_from_accepts_exact_catchup_window() { let db = temp_db("catchup-window-exact"); seed_ordered_txs(db.path.as_str()); - let feed = test_feed(db.path.as_str(), ShutdownSignal::default()); + let feed = test_feed(db.path.as_str(), RuntimeScope::default()); - let subscription = feed.subscribe_from(0, 2); + let subscription = feed.subscribe_from(0, 2).await; assert!( subscription.is_ok(), @@ -96,13 +97,80 @@ async fn subscribe_from_accepts_exact_catchup_window() { ); } +#[test] +fn cancelled_catchup_prepare_retains_process_lock_until_blocking_read_finishes() { + let db = temp_db("cancelled-catchup-prepare-lock"); + seed_ordered_txs(db.path.as_str()); + let data_dir = db._dir.path().to_str().expect("utf8 data dir").to_string(); + let db_path = db.path.clone(); + let runtime = tokio::runtime::Builder::new_current_thread() + .max_blocking_threads(1) + .enable_all() + .build() + .expect("build test runtime"); + + runtime.block_on(async move { + let process_lock = ProcessLock::acquire(&data_dir).expect("acquire process lock"); + let feed = test_feed(&db_path, RuntimeScope::new(process_lock)); + + // Occupy the only blocking thread so subscription preparation is + // deterministically queued, then cancel the async task awaiting it. + let (blocker_started_tx, blocker_started_rx) = oneshot::channel(); + let (release_blocker_tx, release_blocker_rx) = std::sync::mpsc::channel(); + let blocker = tokio::task::spawn_blocking(move || { + let _ = blocker_started_tx.send(()); + release_blocker_rx.recv().expect("release blocking pool"); + }); + blocker_started_rx.await.expect("blocking pool occupied"); + + let (subscribe_entered_tx, subscribe_entered_rx) = oneshot::channel(); + let subscribe = tokio::spawn(async move { + let _ = subscribe_entered_tx.send(()); + feed.subscribe_from(0, u64::MAX).await + }); + subscribe_entered_rx + .await + .expect("subscription preparation entered"); + subscribe.abort(); + let join = match subscribe.await { + Ok(_) => panic!("subscription task should be cancelled"), + Err(join) => join, + }; + assert!(join.is_cancelled()); + + assert!( + matches!( + ProcessLock::acquire(&data_dir), + Err(ProcessLockError::Locked { .. }) + ), + "detached catch-up preparation must retain process ownership" + ); + + release_blocker_tx.send(()).expect("release blocking pool"); + blocker.await.expect("join blocking-pool occupant"); + + let reacquired = tokio::time::timeout(Duration::from_secs(1), async { + loop { + match ProcessLock::acquire(&data_dir) { + Ok(lock) => break lock, + Err(ProcessLockError::Locked { .. }) => tokio::task::yield_now().await, + Err(error) => panic!("unexpected lock acquisition failure: {error}"), + } + } + }) + .await + .expect("detached catch-up preparation should release ownership"); + drop(reacquired); + }); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn subscription_replays_existing_rows_in_order() { let db = temp_db("replay-existing"); seed_ordered_txs(db.path.as_str()); - let feed = test_feed(db.path.as_str(), ShutdownSignal::default()); + let feed = test_feed(db.path.as_str(), RuntimeScope::default()); - let mut subscription = feed.subscribe_from(0, u64::MAX).expect("subscribe"); + let mut subscription = feed.subscribe_from(0, u64::MAX).await.expect("subscribe"); let first = tokio::time::timeout(Duration::from_secs(1), subscription.recv()) .await @@ -145,15 +213,15 @@ async fn subscription_filters_batch_submitter_safe_inputs() { seed_ordered_txs_with_sender(db.path.as_str(), batch_submitter_address); let feed = L2TxFeed::new( db.path.clone(), - ShutdownSignal::default(), + RuntimeScope::default(), L2TxFeedConfig { idle_poll_interval: Duration::from_millis(2), page_size: 64, - batch_submitter_address: Some(batch_submitter_address), + ..L2TxFeedConfig::new(batch_submitter_address) }, ); - let mut subscription = feed.subscribe_from(0, u64::MAX).expect("subscribe"); + let mut subscription = feed.subscribe_from(0, u64::MAX).await.expect("subscribe"); let first = tokio::time::timeout(Duration::from_secs(1), subscription.recv()) .await .expect("wait first event") @@ -179,10 +247,13 @@ async fn subscription_filters_batch_submitter_safe_inputs() { async fn shutdown_signal_closes_subscription() { let db = temp_db("shutdown-closes"); seed_ordered_txs(db.path.as_str()); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let feed = test_feed(db.path.as_str(), shutdown.clone()); - let mut subscription = feed.subscribe_from(u64::MAX, u64::MAX).expect("subscribe"); + let mut subscription = feed + .subscribe_from(u64::MAX, u64::MAX) + .await + .expect("subscribe"); shutdown.request_shutdown(); @@ -195,6 +266,71 @@ async fn shutdown_signal_closes_subscription() { subscription.finish().await.expect("clean shutdown"); } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn terminal_fault_discards_already_queued_subscription_events() { + let db = temp_db("terminal-discards-queued-feed"); + seed_ordered_txs(db.path.as_str()); + let shutdown = RuntimeScope::default(); + let feed = test_feed(db.path.as_str(), shutdown.clone()); + let mut subscription = feed.subscribe_from(0, u64::MAX).await.expect("subscribe"); + tokio::time::sleep(Duration::from_millis(20)).await; + + shutdown.contain_storage_invariant_failure("test fault"); + + assert!( + subscription.recv().await.is_none(), + "biased shutdown must outrank a replay event queued before terminal publication" + ); + subscription + .finish() + .await + .expect("clean terminal shutdown"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn corrupt_feed_head_trips_terminal_storage_fault() { + let db = temp_db("corrupt-feed-head"); + seed_ordered_txs(db.path.as_str()); + let conn = Storage::open_connection(db.path.as_str()).expect("raw connection"); + conn.execute("UPDATE sequenced_l2_txs SET offset = -offset", []) + .expect("corrupt offsets"); + drop(conn); + + let shutdown = RuntimeScope::default(); + let feed = test_feed(db.path.as_str(), shutdown.clone()); + + assert!(matches!( + feed.subscribe_from(0, u64::MAX).await, + Err(SubscribeError::StorageInvariantViolation) + )); + assert!(shutdown.is_storage_invariant_contained()); + assert!(shutdown.is_shutdown_requested()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn corrupt_feed_page_trips_terminal_storage_fault() { + let db = temp_db("corrupt-feed-page"); + seed_ordered_txs(db.path.as_str()); + let conn = Storage::open_connection(db.path.as_str()).expect("raw connection"); + conn.execute("UPDATE frames SET safe_block = 'not-an-integer'", []) + .expect("corrupt safe-block storage type"); + drop(conn); + + let shutdown = RuntimeScope::default(); + let feed = test_feed(db.path.as_str(), shutdown.clone()); + let subscription = feed.subscribe_from(0, u64::MAX).await.expect("subscribe"); + + tokio::time::timeout(Duration::from_secs(1), shutdown.wait_for_shutdown()) + .await + .expect("terminal fault containment requests shutdown"); + assert!(shutdown.is_storage_invariant_contained()); + assert!(shutdown.is_shutdown_requested()); + assert!(matches!( + subscription.finish().await, + Err(SubscriptionError::StorageInvariantViolation) + )); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn catchup_window_not_inflated_by_invalidated_batch_holes() { // Regression test: after batch invalidation, offset holes in sequenced_l2_txs @@ -255,9 +391,9 @@ async fn catchup_window_not_inflated_by_invalidated_batch_holes() { // Before invalidation: 2 valid events. // With max_catchup_events=1, subscribing from 0 should fail. - let feed = test_feed(db.path.as_str(), ShutdownSignal::default()); + let feed = test_feed(db.path.as_str(), RuntimeScope::default()); assert!( - feed.subscribe_from(0, 1).is_err(), + feed.subscribe_from(0, 1).await.is_err(), "should reject: 2 valid events > max 1" ); @@ -268,9 +404,9 @@ async fn catchup_window_not_inflated_by_invalidated_batch_holes() { drop(storage); // After invalidation: only 1 valid event, so max_catchup_events=1 should succeed. - let feed = test_feed(db.path.as_str(), ShutdownSignal::default()); + let feed = test_feed(db.path.as_str(), RuntimeScope::default()); assert!( - feed.subscribe_from(0, 1).is_ok(), + feed.subscribe_from(0, 1).await.is_ok(), "should accept: only 1 valid event after invalidation, despite rowid hole" ); } @@ -320,43 +456,42 @@ async fn catchup_window_excludes_batch_submitter_direct_inputs() { .expect("close frame"); drop(storage); - // Without batch_submitter_address filtering: 2 events, max=1 should reject. + // With a submitter address that matches no seeded sender: 2 events, + // max=1 should reject. let feed_no_filter = L2TxFeed::new( db.path.clone(), - ShutdownSignal::default(), - L2TxFeedConfig { - batch_submitter_address: None, - ..L2TxFeedConfig::default() - }, + RuntimeScope::default(), + L2TxFeedConfig::new(NO_OWN_BATCHES), ); assert!( - feed_no_filter.subscribe_from(0, 1).is_err(), + feed_no_filter.subscribe_from(0, 1).await.is_err(), "without filter: 2 events > max 1" ); // With batch_submitter_address filtering: only the user's event counts. let feed_filtered = L2TxFeed::new( db.path.clone(), - ShutdownSignal::default(), - L2TxFeedConfig { - batch_submitter_address: Some(batch_submitter), - ..L2TxFeedConfig::default() - }, + RuntimeScope::default(), + L2TxFeedConfig::new(batch_submitter), ); assert!( - feed_filtered.subscribe_from(0, 1).is_ok(), + feed_filtered.subscribe_from(0, 1).await.is_ok(), "with filter: only 1 broadcastable event, should accept" ); } -fn test_feed(db_path: &str, shutdown: ShutdownSignal) -> L2TxFeed { +/// Sentinel submitter for fixtures that seed no own-batch rows. Must not +/// collide with any seeded sender (`seed_ordered_txs` uses `Address::ZERO`). +const NO_OWN_BATCHES: Address = Address::repeat_byte(0x7f); + +fn test_feed(db_path: &str, shutdown: RuntimeScope) -> L2TxFeed { L2TxFeed::new( db_path.to_string(), shutdown, L2TxFeedConfig { idle_poll_interval: Duration::from_millis(2), page_size: 64, - batch_submitter_address: None, + ..L2TxFeedConfig::new(NO_OWN_BATCHES) }, ) } diff --git a/sequencer/src/harness.rs b/sequencer/src/harness.rs index f1b97a67..d359800b 100644 --- a/sequencer/src/harness.rs +++ b/sequencer/src/harness.rs @@ -1,13 +1,13 @@ // (c) Cartesi and individual authors (see AUTHORS) // SPDX-License-Identifier: Apache-2.0 (see LICENSE) -//! CLI harness: the subcommand parser, dispatch, and R4 exit-code projection, +//! CLI harness: the subcommand parser, dispatch, and exit-code projection, //! exported once from the library so every app binary inherits them. //! //! An app's `main` is ~5 lines: init tracing, then [`run_main`] with a -//! genesis-app factory closure. The factory is only invoked by `setup` (to -//! write the genesis finalized snapshot); `run` and `flush-mempool` never -//! construct an app value. +//! genesis-app factory closure. The factory is invoked only when plain `setup` +//! reaches genesis-snapshot registration; completed setup no-ops, recovery, +//! `run`, and `flush-mempool` never construct an app value. //! //! ```ignore //! #[tokio::main] @@ -26,7 +26,7 @@ use clap::{Parser, Subcommand}; use sequencer_core::application::Application; -use crate::runtime::config::{FlushConfig, RunConfig, SetupConfig}; +use crate::commands::config::{FlushConfig, RunConfig, SetupConfig}; /// Top-level CLI. Apps parse this (via [`run_main`]) and dispatch. #[derive(Debug, Parser)] @@ -60,30 +60,55 @@ pub enum Command { FlushMempool(Box), } -/// Parse argv and dispatch. Returns the R4 process exit code. +/// Parse argv and dispatch. Returns the process exit code. pub async fn run_main(genesis_app: F) -> std::process::ExitCode where A: Application + Clone + Sync + 'static, - F: FnOnce() -> A, + F: FnOnce() -> A + Send + 'static, { let cli = Cli::parse(); - dispatch(cli.command, genesis_app).await + project_dispatch_join(tokio::spawn(dispatch(cli.command, genesis_app)).await) } -/// Dispatch a parsed [`Command`], projecting the result onto the R4 exit-code -/// contract (see [`crate::runtime::error`]). Clean completion is exit 0; every -/// `RunError` maps through `RunError::exit_code`. +fn project_dispatch_join( + result: Result, +) -> std::process::ExitCode { + match result { + Ok(code) => code, + Err(join) if join.is_panic() => { + tracing::error!( + error = %join, + exit_code = crate::commands::error::EXIT_TERMINAL, + "sequencer command panicked — trusted-code invariant failure" + ); + std::process::ExitCode::from(crate::commands::error::EXIT_TERMINAL) + } + Err(join) => { + tracing::error!( + error = %join, + exit_code = crate::commands::error::EXIT_UNCLASSIFIED, + "sequencer command task failed" + ); + std::process::ExitCode::from(crate::commands::error::EXIT_UNCLASSIFIED) + } + } +} + +/// Dispatch a parsed [`Command`], projecting the result onto the exit-code +/// contract (see [`crate::commands::error`]). Clean completion is exit 0; every +/// `CommandError` maps through `CommandError::exit_code`. /// -/// `genesis_app` is called at most once — only by `setup`. +/// `genesis_app` is called at most once — only when plain `setup` needs to +/// register the genesis snapshot. pub async fn dispatch(command: Command, genesis_app: F) -> std::process::ExitCode where A: Application + Clone + Sync + 'static, F: FnOnce() -> A, { let result = match command { - Command::Setup(config) => crate::runtime::setup::setup(*config, genesis_app()).await, - Command::Run(config) => crate::runtime::run::(*config).await, - Command::FlushMempool(config) => crate::runtime::flush::flush_mempool(*config).await, + Command::Setup(config) => crate::commands::setup::setup(*config, genesis_app).await, + Command::Run(config) => crate::commands::run::run::(*config).await, + Command::FlushMempool(config) => crate::commands::flush::flush_mempool(*config).await, }; match result { @@ -95,3 +120,78 @@ where } } } + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn completed_setup_does_not_construct_genesis_app() { + use crate::commands::test_support::SweepTestApp; + use crate::storage::{LifecycleCommand, Storage}; + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + + let data_dir = tempfile::tempdir().expect("create data dir"); + let data_dir = data_dir.path().to_string_lossy().into_owned(); + let cli = Cli::try_parse_from([ + "sequencer", + "setup", + "--data-dir", + data_dir.as_str(), + "--eth-rpc-url", + "http://127.0.0.1:1", + "--chain-id", + "31337", + "--app-address", + "0x1111111111111111111111111111111111111111", + "--batch-submitter-address", + "0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266", + ]) + .expect("parse setup"); + let db_path = match &cli.command { + Command::Setup(config) => config.db_path(), + other => panic!("expected setup subcommand, got {other:?}"), + }; + + let mut storage = Storage::initialize_for_command(&db_path, LifecycleCommand::Setup) + .expect("initialize setup"); + storage + .insert_initial_finalized_dump( + &std::path::Path::new(&data_dir).join("finalized"), + 0, + 0, + 0, + 0, + ) + .expect("register finalized snapshot"); + storage.complete_setup().expect("complete setup"); + drop(storage); + + let constructions = Arc::new(AtomicUsize::new(0)); + let observed = Arc::clone(&constructions); + let exit = dispatch::(cli.command, move || { + observed.fetch_add(1, Ordering::SeqCst); + SweepTestApp + }) + .await; + + assert_eq!(exit, std::process::ExitCode::SUCCESS); + assert_eq!(constructions.load(Ordering::SeqCst), 0); + } + + #[tokio::test] + async fn top_level_command_panic_maps_to_terminal_exit() { + let result = tokio::spawn(async { + panic!("trusted-code invariant failure"); + #[allow(unreachable_code)] + std::process::ExitCode::SUCCESS + }) + .await; + + assert_eq!( + project_dispatch_join(result), + std::process::ExitCode::from(crate::commands::error::EXIT_TERMINAL) + ); + } +} diff --git a/sequencer/src/http.rs b/sequencer/src/http.rs index 264a9ed8..d0ff0a8e 100644 --- a/sequencer/src/http.rs +++ b/sequencer/src/http.rs @@ -9,7 +9,6 @@ //! side on its own port (same binary, two listeners). When that lands, the //! orchestration here becomes per-side `start_*` calls. -use std::io; use std::sync::Arc; use alloy_sol_types::Eip712Domain; @@ -21,6 +20,7 @@ use axum::response::{IntoResponse, Response}; use serde::Serialize; use thiserror::Error; use tokio::sync::mpsc; +use tokio::task::{JoinHandle, JoinSet}; use tower_http::cors::CorsLayer; use tower_http::trace::TraceLayer; @@ -29,7 +29,8 @@ use crate::egress::api::SubscribeState; use crate::egress::l2_tx_feed::L2TxFeed; use crate::ingress::api::SubmitState; use crate::ingress::inclusion_lane::{PendingUserOp, SequencerError}; -use crate::runtime::shutdown::ShutdownSignal; +use crate::runtime::shutdown::RuntimeScope; +use crate::storage::ReleaseScheduler; use sequencer_core::api::{TxRequest, TxRequestError}; #[derive(Debug, Error, Clone)] @@ -151,18 +152,120 @@ const DEFAULT_MAX_BODY_BYTES: usize = TxRequest::MAX_JSON_BYTES_RECOMMENDED; /// The full reason is `{WS_CATCHUP_WINDOW_EXCEEDED_REASON}: live_start_offset=`. pub const WS_CATCHUP_WINDOW_EXCEEDED_REASON: &str = "catch-up window exceeded"; -pub type ApiServerTask = tokio::task::JoinHandle>; +pub type ApiServerTask = JoinHandle>; -#[derive(Debug, Clone, Copy)] +type SnapshotReleaseTask = Box; + +/// Joins every blocking snapshot-lease release before the HTTP worker exits. +/// +/// The scheduler passed into each lease guard owns a channel producer. The +/// receiver therefore cannot close while a guard can still submit a release, +/// including a guard dropping concurrently with graceful HTTP shutdown. +struct SnapshotReleaseDrain { + supervisor: JoinHandle<()>, + shutdown: RuntimeScope, +} + +impl SnapshotReleaseDrain { + async fn finish(self) { + if let Err(join) = self.supervisor.await { + tracing::error!( + error = %join, + "snapshot lease release supervisor task failed" + ); + self.shutdown.contain_storage_invariant_failure(format!( + "snapshot lease release supervisor task failed: {join}" + )); + } + } +} + +fn supervise_snapshot_releases(shutdown: RuntimeScope) -> (ReleaseScheduler, SnapshotReleaseDrain) { + let (sender, receiver) = mpsc::unbounded_channel::(); + let schedule_shutdown = shutdown.clone(); + let scheduler: ReleaseScheduler = Arc::new(move |release| { + if sender.send(release).is_err() { + tracing::error!("snapshot lease release supervisor is unavailable"); + // A closed receiver while this producer still exists means the + // supervisor failed. Containment is sync and callable from any + // thread — the old `tokio::spawn` wrapper was residue of the + // deleted async containment API and would have panicked on a + // non-runtime thread. `SnapshotReleaseDrain` also + // classifies its join before the HTTP worker can finish. + schedule_shutdown.contain_storage_invariant_failure( + "snapshot lease release supervisor is unavailable", + ); + } + }); + let supervisor_shutdown = shutdown.clone(); + let supervisor = tokio::spawn(async move { + run_snapshot_release_supervisor(receiver, supervisor_shutdown).await; + }); + ( + scheduler, + SnapshotReleaseDrain { + supervisor, + shutdown, + }, + ) +} + +async fn run_snapshot_release_supervisor( + mut receiver: mpsc::UnboundedReceiver, + shutdown: RuntimeScope, +) { + let mut releases = JoinSet::new(); + let mut accepting = true; + + while accepting || !releases.is_empty() { + tokio::select! { + task = receiver.recv(), if accepting => { + match task { + Some(release) => { + releases.spawn_blocking(release); + } + None => accepting = false, + } + } + result = releases.join_next(), if !releases.is_empty() => { + if let Some(Err(join)) = result { + tracing::error!( + error = %join, + "snapshot lease release task failed" + ); + shutdown + .contain_storage_invariant_failure(format!( + "snapshot lease release task panicked: {join}" + )); + } + } + } + } +} + +/// The API's per-deployment configuration: the two ingress values that vary +/// (the EIP-712 verification domain and the app's payload bound) plus three +/// service limits. The limits are module constants by design — not +/// operator-tunable, no CLI flags owed; they are fields only so tests can +/// narrow them. +#[derive(Debug, Clone)] pub struct ApiConfig { + /// EIP-712 domain user-op signatures are verified against. + pub domain: Eip712Domain, + /// The app's `MAX_METHOD_PAYLOAD_BYTES` bound on user-op payloads. + pub max_user_op_data_bytes: usize, pub max_body_bytes: usize, pub ws_max_subscribers: usize, pub ws_max_catchup_events: u64, } -impl Default for ApiConfig { - fn default() -> Self { +impl ApiConfig { + /// The only constructor: the deployment-varying values are mandatory, + /// the service limits take the module defaults. + pub fn new(domain: Eip712Domain, max_user_op_data_bytes: usize) -> Self { Self { + domain, + max_user_op_data_bytes, max_body_bytes: DEFAULT_MAX_BODY_BYTES, ws_max_subscribers: DEFAULT_WS_MAX_SUBSCRIBERS, ws_max_catchup_events: DEFAULT_WS_MAX_CATCHUP_EVENTS, @@ -170,49 +273,24 @@ impl Default for ApiConfig { } } -#[allow(clippy::too_many_arguments)] -pub async fn start( - http_addr: impl tokio::net::ToSocketAddrs, - tx_sender: mpsc::Sender, - domain: Eip712Domain, - max_user_op_data_bytes: usize, - shutdown: ShutdownSignal, - tx_feed: L2TxFeed, - config: ApiConfig, - snapshot_state: SnapshotState, -) -> io::Result { - let listener = tokio::net::TcpListener::bind(http_addr).await?; - Ok(start_on_listener( - listener, - tx_sender, - domain, - max_user_op_data_bytes, - shutdown, - tx_feed, - config, - snapshot_state, - )) -} - -#[allow(clippy::too_many_arguments)] -pub fn start_on_listener( +pub(crate) fn start_on_listener( listener: tokio::net::TcpListener, tx_sender: mpsc::Sender, - domain: Eip712Domain, - max_user_op_data_bytes: usize, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, tx_feed: L2TxFeed, config: ApiConfig, snapshot_state: SnapshotState, ) -> ApiServerTask { + let (snapshot_release_scheduler, snapshot_release_drain) = + supervise_snapshot_releases(shutdown.clone()); let health_state = Arc::new(crate::egress::api::HealthState { tx_sender: tx_sender.clone(), shutdown: shutdown.clone(), }); let submit_state = Arc::new(SubmitState::new( tx_sender, - domain, - max_user_op_data_bytes, + config.domain, + config.max_user_op_data_bytes, shutdown.clone(), )); let subscribe_state = Arc::new(SubscribeState::new( @@ -225,7 +303,9 @@ pub fn start_on_listener( .merge(crate::egress::api::router( subscribe_state, health_state, - Arc::new(snapshot_state), + snapshot_state, + shutdown.clone(), + snapshot_release_scheduler, )) // Enforces a raw request-body cap before JSON deserialization, including whitespace. .layer(DefaultBodyLimit::max(config.max_body_bytes)) @@ -235,10 +315,73 @@ pub fn start_on_listener( .layer(CorsLayer::permissive()); tokio::spawn(async move { - axum::serve(listener, app) + let result = axum::serve(listener, app) .with_graceful_shutdown(async move { shutdown.wait_for_shutdown().await; }) - .await + .await; + // `Workers::finish` awaits this server handle before its final sticky + // fault check, so no supervised release may outlive classification. + snapshot_release_drain.finish().await; + result }) } + +#[cfg(test)] +mod tests { + use std::time::Duration; + + use tokio::sync::oneshot; + + use super::*; + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn snapshot_release_drain_waits_for_late_terminal_report() { + let shutdown = RuntimeScope::default(); + let (schedule, drain) = supervise_snapshot_releases(shutdown.clone()); + let (started_tx, started_rx) = oneshot::channel(); + let (unblock_tx, unblock_rx) = std::sync::mpsc::channel(); + let release_shutdown = shutdown.clone(); + let mut drain_task = tokio::spawn(drain.finish()); + + // Start draining first. The scheduler's producer token must keep the + // queue open so this release, submitted concurrently with shutdown, + // cannot be missed. + tokio::task::yield_now().await; + assert!(!drain_task.is_finished()); + + schedule(Box::new(move || { + let _ = started_tx.send(()); + unblock_rx.recv().expect("release test gate"); + release_shutdown.contain_storage_invariant_failure("test release fault"); + })); + started_rx.await.expect("release task started"); + drop(schedule); + + assert!( + tokio::time::timeout(Duration::from_millis(25), &mut drain_task) + .await + .is_err(), + "drain returned before the in-flight release completed" + ); + + unblock_tx.send(()).expect("unblock release"); + drain_task.await.expect("join release drain"); + assert!( + shutdown.is_storage_invariant_contained(), + "the terminal report must be sticky before the drain returns" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn snapshot_release_task_panic_is_terminal_before_drain_returns() { + let shutdown = RuntimeScope::default(); + let (schedule, drain) = supervise_snapshot_releases(shutdown.clone()); + + schedule(Box::new(|| panic!("simulated lease release panic"))); + drop(schedule); + drain.finish().await; + + assert!(shutdown.is_storage_invariant_contained()); + } +} diff --git a/sequencer/src/ingress/api.rs b/sequencer/src/ingress/api.rs index 9ed754b8..e1ce82c2 100644 --- a/sequencer/src/ingress/api.rs +++ b/sequencer/src/ingress/api.rs @@ -12,6 +12,7 @@ use alloy_sol_types::Eip712Domain; use axum::Router; use axum::extract::{Json, State}; use axum::http::StatusCode; +use axum::response::{IntoResponse, Response}; use axum::routing::post; use tokio::sync::mpsc::{self, error::TrySendError}; use tokio::sync::oneshot; @@ -19,7 +20,7 @@ use tracing::debug; use crate::http::ApiError; use crate::ingress::inclusion_lane::PendingUserOp; -use crate::runtime::shutdown::ShutdownSignal; +use crate::runtime::shutdown::RuntimeScope; use sequencer_core::api::{TxRequest, TxResponse}; use sequencer_core::user_op::SignedUserOp; @@ -29,7 +30,7 @@ pub(crate) struct SubmitState { pub tx_sender: mpsc::Sender, pub domain: Eip712Domain, pub max_user_op_data_bytes: usize, - pub shutdown: ShutdownSignal, + pub shutdown: RuntimeScope, } impl SubmitState { @@ -37,7 +38,7 @@ impl SubmitState { tx_sender: mpsc::Sender, domain: Eip712Domain, max_user_op_data_bytes: usize, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, ) -> Self { Self { tx_sender, @@ -66,7 +67,7 @@ pub(crate) fn router(state: Arc) -> Router { async fn submit_tx( State(state): State>, req: Result, axum::extract::rejection::JsonRejection>, -) -> Result, ApiError> { +) -> Result { let Json(req) = req.map_err(map_json_rejection)?; let signed = req @@ -80,13 +81,20 @@ async fn submit_tx( .await .map_err(|_| ApiError::internal_error("inclusion lane dropped response"))?; commit_result.map_err(ApiError::from)?; + // Publication gate: the lane's acknowledgement already required the + // token; the success body after a post-commit containment is suppressed + // by the same consult. + if state.shutdown.authorize().is_none() { + return Err(ApiError::unavailable("sequencer shutting down")); + } debug!(sender = %sender, nonce, "tx committed"); Ok(Json(TxResponse { ok: true, sender: sender.to_string(), nonce, - })) + }) + .into_response()) } /// Normalize JSON-extractor failures into fixed client-facing messages. @@ -126,7 +134,7 @@ fn enqueue_verified_tx( match state.tx_sender.try_send(pending) { Ok(()) => Ok(recv), Err(TrySendError::Full(_)) => Err(ApiError::overloaded("queue full")), - Err(TrySendError::Closed(_)) => Err(ApiError::internal_error("inclusion lane unavailable")), + Err(TrySendError::Closed(_)) => Err(ApiError::unavailable("inclusion lane unavailable")), } } @@ -147,12 +155,40 @@ mod tests { use crate::storage::Storage; use sequencer_core::user_op::UserOp; + #[test] + fn closed_lane_is_service_unavailable() { + let (tx_sender, rx) = mpsc::channel::(1); + drop(rx); + let state = SubmitState::new( + tx_sender, + Eip712Domain::default(), + 128, + RuntimeScope::default(), + ); + let signed = SignedUserOp { + sender: Address::ZERO, + signature: Signature::test_signature(), + user_op: UserOp { + nonce: 0, + max_fee: 0, + data: Vec::new().into(), + }, + }; + + let err = match enqueue_verified_tx(&state, signed) { + Ok(_) => panic!("closed lane must reject admission"), + Err(err) => err, + }; + assert_eq!(err.status(), StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(err.code(), "UNAVAILABLE"); + } + #[tokio::test(flavor = "current_thread")] async fn submit_tx_rejects_when_shutdown_has_started() { let db = TempDir::new().expect("create temp dir"); let db_path = db.path().join("sequencer.db"); let _storage = Storage::open(&db_path.to_string_lossy()).expect("create db"); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); shutdown.request_shutdown(); let (tx_sender, _rx) = mpsc::channel::(1); @@ -189,6 +225,54 @@ mod tests { assert_eq!(err.code(), "UNAVAILABLE"); } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn terminal_fault_after_enqueue_prevents_success_response() { + let db = TempDir::new().expect("create temp dir"); + let db_path = db.path().join("sequencer.db"); + let _storage = Storage::open(&db_path.to_string_lossy()).expect("create db"); + let shutdown = RuntimeScope::default(); + let (tx_sender, mut rx) = mpsc::channel::(1); + let state = Arc::new(SubmitState::new( + tx_sender, + Eip712Domain { + name: None, + version: None, + chain_id: None, + verifying_contract: None, + salt: None, + }, + 128, + shutdown.clone(), + )); + let signing_key = SigningKey::from_bytes((&[7_u8; 32]).into()).expect("create signing key"); + let sender = address_from_signing_key(&signing_key); + let user_op = UserOp { + nonce: 0, + max_fee: 0, + data: Vec::new().into(), + }; + let request = TxRequest { + message: user_op.clone(), + signature: sign_user_op_hex(&state.domain, &user_op, &signing_key), + sender: sender.to_string(), + }; + let response = tokio::spawn(submit_tx(State(state), Ok(Json(request)))); + let pending = rx.recv().await.expect("request reached the lane"); + + shutdown.contain_storage_invariant_failure("test fault"); + pending + .respond_to + .send(Ok(())) + .expect("simulate a stale post-fault lane acknowledgement"); + + let err = response + .await + .expect("handler task") + .expect_err("terminal publication must prevent HTTP 200"); + assert_eq!(err.status(), StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(err.code(), "UNAVAILABLE"); + } + fn sign_user_op_hex( domain: &Eip712Domain, user_op: &UserOp, diff --git a/sequencer/src/ingress/inclusion_lane/catch_up.rs b/sequencer/src/ingress/inclusion_lane/catch_up.rs index 62a22a3e..603f58d6 100644 --- a/sequencer/src/ingress/inclusion_lane/catch_up.rs +++ b/sequencer/src/ingress/inclusion_lane/catch_up.rs @@ -10,7 +10,8 @@ use std::path::PathBuf; use alloy_primitives::Address; use crate::storage::Storage; -use sequencer_core::application::Application; +use sequencer_core::application::{Application, execute_direct_input, execute_valid_user_op}; +use sequencer_core::history::ExecutedInputCount; use sequencer_core::l2_tx::SequencedL2Tx; use super::error::CatchUpError; @@ -27,6 +28,7 @@ const DEFAULT_CATCH_UP_PAGE_SIZE: usize = 256; pub(super) struct CatchUpSnapshot { pub(super) dump_dir: PathBuf, pub(super) l2_tx_index: u64, + pub(super) executed_input_count: ExecutedInputCount, } /// Select the resume checkpoint. Prefers the latest pending snapshot @@ -36,20 +38,21 @@ pub(super) struct CatchUpSnapshot { /// back to the finalized snapshot. /// /// Returns the checkpoint directly rather than an `Option`: the -/// always-load invariant — `setup` registers the genesis finalized -/// snapshot, `run` gates on it (`require_finalized_snapshot` in -/// `Workers::spawn`, plus the boot-gate check) before `InclusionLane::start` -/// — guarantees at least the genesis finalized snapshot exists by the time +/// always-load invariant — `setup` registers the genesis finalized snapshot, +/// and `run` checks it in reducer inspection plus task-free runtime +/// preparation before `InclusionLane::start` — guarantees at least the +/// genesis finalized snapshot exists by the time /// the lane resumes. Absence is a violated invariant (runtime/setup bug), /// surfaced fail-loud as [`CatchUpError::NoSnapshot`]. pub(super) fn catch_up_snapshot(storage: &mut Storage) -> Result { - let (dump, l2_tx_index) = storage + let (dump, l2_tx_index, executed_input_count) = storage .latest_snapshot() .map_err(|source| CatchUpError::LoadSnapshot { source })? .ok_or(CatchUpError::NoSnapshot)?; Ok(CatchUpSnapshot { dump_dir: dump.prefix, l2_tx_index, + executed_input_count, }) } @@ -96,8 +99,16 @@ pub(super) fn catch_up_application_paged( return Ok(()); } - for (db_offset, item, frame_safe_block) in replay { - replay_sequenced_l2_tx(app, batch_submitter_address, item, frame_safe_block)?; + for row in replay { + let db_offset = row.db_offset; + replay_sequenced_l2_tx( + app, + batch_submitter_address, + db_offset, + row.tx, + row.frame_safe_block, + row.executed_input_offset, + )?; next_offset = db_offset; } } @@ -106,30 +117,63 @@ pub(super) fn catch_up_application_paged( fn replay_sequenced_l2_tx( app: &mut impl Application, batch_submitter_address: Address, + db_offset: u64, item: SequencedL2Tx, frame_safe_block: u64, + executed_input_offset: Option, ) -> Result<(), CatchUpError> { match item { SequencedL2Tx::UserOp(value) => { + assert_replay_mapping( + db_offset, + "user op", + Some(app.executed_input_count()), + executed_input_offset, + )?; // The persisted covering frame's safe_block mirrors what the // lane passed live, so the replayed app's safe-block clock // lands on the same value. - app.execute_valid_user_op(&value, frame_safe_block) + execute_valid_user_op(app, &value, frame_safe_block) .map(|_| ()) - .map_err(|err| CatchUpError::ReplayUserOpInternal { - reason: err.to_string(), - }) + .map_err(|source| CatchUpError::ReplayUserOp { source }) } SequencedL2Tx::Direct(direct) => { if direct.sender == batch_submitter_address { + assert_replay_mapping( + db_offset, + "batch-submitter input", + None, + executed_input_offset, + )?; return Ok(()); } - app.execute_direct_input(&direct) + assert_replay_mapping( + db_offset, + "direct input", + Some(app.executed_input_count()), + executed_input_offset, + )?; + execute_direct_input(app, &direct) .map(|_| ()) - .map_err(|err| CatchUpError::ReplayDirectInputInternal { - reason: err.to_string(), - }) + .map_err(|source| CatchUpError::ReplayDirectInput { source }) } } } + +fn assert_replay_mapping( + db_offset: u64, + kind: &'static str, + expected: Option, + stored: Option, +) -> Result<(), CatchUpError> { + if expected == stored { + return Ok(()); + } + Err(CatchUpError::ExecutionOffsetMismatch { + db_offset, + kind, + expected: expected.map(ExecutedInputCount::get), + stored: stored.map(ExecutedInputCount::get), + }) +} diff --git a/sequencer/src/ingress/inclusion_lane/config.rs b/sequencer/src/ingress/inclusion_lane/config.rs index 24f139b7..276654f0 100644 --- a/sequencer/src/ingress/inclusion_lane/config.rs +++ b/sequencer/src/ingress/inclusion_lane/config.rs @@ -14,8 +14,8 @@ const DEFAULT_SAFE_INPUT_BUFFER_CAPACITY: usize = 2048; const DEFAULT_MAX_BATCH_OPEN: Duration = Duration::from_secs(2 * 60 * 60); const DEFAULT_IDLE_POLL_INTERVAL: Duration = Duration::from_millis(10); /// Minimum gap between L1 safe-frontier polls. Bounds the SQL load when the -/// lane is otherwise idle. L1 safe head advances at ~12s cadence, so 1s is -/// well inside the responsiveness budget. +/// lane is otherwise idle. Safe-head observations advance materially less +/// often than user-op chunks, so 1s is inside the reconciliation budget. const DEFAULT_FRONTIER_MIN_INTERVAL: Duration = Duration::from_secs(1); #[derive(Debug, Clone)] @@ -31,8 +31,9 @@ pub struct InclusionLaneConfig { /// Cap on user ops dequeued per chunk. Bounds per-chunk SQL transaction /// size and (more importantly) ack latency for the first op in each chunk. pub max_user_ops_per_chunk: usize, - /// Reusable buffer size for safe-input loading. Doesn't bound work; just - /// the memory ceiling for the read-and-execute scratch buffer. + /// Reusable buffer size for safe-input loading. It bounds scratch memory, + /// not work or time: one reconciliation turn consumes the complete + /// newly-safe range without timeout/resume state. pub safe_input_buffer_capacity: usize, /// Force a batch close after this much wall time, regardless of size. pub max_batch_open: Duration, diff --git a/sequencer/src/ingress/inclusion_lane/dump_info.rs b/sequencer/src/ingress/inclusion_lane/dump_info.rs index 6809e4bd..56b5aa41 100644 --- a/sequencer/src/ingress/inclusion_lane/dump_info.rs +++ b/sequencer/src/ingress/inclusion_lane/dump_info.rs @@ -43,6 +43,20 @@ pub fn app_prefix(dump_dir: &Path) -> PathBuf { dump_dir.join(APP_STATE_SUBDIR) } +/// Whether an I/O error proves that a DB-referenced snapshot artifact is +/// missing or structurally corrupt. Other filesystem failures remain +/// operational: they may clear when the device, mount, or permissions recover. +pub(crate) fn referenced_artifact_io_is_terminal(source: &io::Error) -> bool { + matches!( + source.kind(), + io::ErrorKind::NotFound + | io::ErrorKind::InvalidData + | io::ErrorKind::UnexpectedEof + | io::ErrorKind::NotADirectory + | io::ErrorKind::IsADirectory + ) +} + /// Sequencer-owned checkpoint metadata for one dump. /// /// Serialized as real TOML (`#[serde(deny_unknown_fields)]` keeps the parse @@ -271,6 +285,26 @@ mod tests { assert_eq!(read_info(dir.path()).unwrap(), sample()); } + #[test] + fn referenced_artifact_classifier_separates_corruption_from_operational_io() { + for kind in [ + io::ErrorKind::NotFound, + io::ErrorKind::InvalidData, + io::ErrorKind::UnexpectedEof, + io::ErrorKind::NotADirectory, + io::ErrorKind::IsADirectory, + ] { + assert!( + referenced_artifact_io_is_terminal(&io::Error::from(kind)), + "{kind:?} proves a referenced artifact is unusable" + ); + } + assert!( + !referenced_artifact_io_is_terminal(&io::Error::other("filesystem unavailable")), + "unclassified filesystem failures remain operational" + ); + } + #[test] fn stamp_fills_b_and_is_idempotent() { let dir = tempfile::tempdir().unwrap(); diff --git a/sequencer/src/ingress/inclusion_lane/error.rs b/sequencer/src/ingress/inclusion_lane/error.rs index dab7f99c..1ca98e51 100644 --- a/sequencer/src/ingress/inclusion_lane/error.rs +++ b/sequencer/src/ingress/inclusion_lane/error.rs @@ -7,6 +7,7 @@ use sequencer_core::application::AppError; use thiserror::Error; +use super::dump_info::CreateDumpDirError; use super::snapshot::{GcError, StampError, TakeDumpError}; #[derive(Debug, Error)] @@ -20,6 +21,13 @@ pub enum InclusionLaneError { }, #[error(transparent)] Storage(#[from] rusqlite::Error), + #[error("terminal storage invariant failure requested runtime shutdown")] + TerminalStorageInvariant, + #[error( + "canonical divergence at batch nonce {nonce}, safe-input index {safe_input_index}; \ + cockroach recovery required" + )] + CanonicalDivergence { nonce: u64, safe_input_index: u64 }, #[error("user op execution failed")] ExecuteUserOp { #[source] @@ -40,11 +48,52 @@ pub enum InclusionLaneError { PromotionStamp(#[from] StampError), #[error( "no open Tip at lane startup; the runtime must establish it via \ - Storage::ensure_open_tip before starting the lane" + the recovery reducer's EnsureOpenTip phase before starting the lane" )] NoOpenTip, } +impl InclusionLaneError { + pub(crate) fn is_terminal_invariant(&self) -> bool { + match self { + Self::Storage(source) => crate::storage::is_persistent_storage_error(source), + Self::CatchUp { source } => source.is_terminal_invariant(), + Self::ExecuteUserOp { source } | Self::ExecuteDirectInput { source } => { + app_error_is_terminal(source) + } + Self::LoadFromDump(source) => referenced_snapshot_app_error_is_terminal(source), + Self::Snapshot(source) => take_dump_error_is_terminal(source), + Self::Gc(GcError::Storage(source)) + | Self::PromotionStamp(StampError::Storage(source)) => { + crate::storage::is_persistent_storage_error(source) + } + Self::TerminalStorageInvariant | Self::CanonicalDivergence { .. } | Self::NoOpenTip => { + true + } + Self::ChannelClosed | Self::PromotionStamp(StampError::Io(_)) => false, + } + } +} + +fn app_error_is_terminal(source: &AppError) -> bool { + matches!(source, AppError::Internal { .. }) +} + +fn referenced_snapshot_app_error_is_terminal(source: &AppError) -> bool { + match source { + AppError::Internal { .. } => true, + AppError::Io(source) => super::dump_info::referenced_artifact_io_is_terminal(source), + } +} + +fn take_dump_error_is_terminal(source: &TakeDumpError) -> bool { + match source { + TakeDumpError::Storage(source) => crate::storage::is_persistent_storage_error(source), + TakeDumpError::CreateDump(CreateDumpDirError::App(source)) => app_error_is_terminal(source), + TakeDumpError::CreateDump(CreateDumpDirError::Io(_)) => false, + } +} + #[derive(Debug, Error)] pub enum CatchUpError { #[error("cannot load resume snapshot")] @@ -58,13 +107,152 @@ pub enum CatchUpError { #[source] source: rusqlite::Error, }, - #[error("replay user op failed: {reason}")] - ReplayUserOpInternal { reason: String }, - #[error("replay direct input failed: {reason}")] - ReplayDirectInputInternal { reason: String }, + #[error("replay user op failed: {source}")] + ReplayUserOp { + #[source] + source: AppError, + }, + #[error("replay direct input failed: {source}")] + ReplayDirectInput { + #[source] + source: AppError, + }, + #[error("snapshot executed-input count mismatch: application={application}, storage={storage}")] + SnapshotExecutionCountMismatch { application: u64, storage: u64 }, + #[error( + "physical replay row {db_offset} ({kind}) has execution offset {stored:?}, expected {expected:?}" + )] + ExecutionOffsetMismatch { + db_offset: u64, + kind: &'static str, + expected: Option, + stored: Option, + }, #[error( "no snapshot registered before lane catch-up; \ runtime must ensure a genesis dump exists at first startup" )] NoSnapshot, } + +impl CatchUpError { + fn is_terminal_invariant(&self) -> bool { + match self { + Self::LoadSnapshot { source } | Self::LoadReplay { source, .. } => { + crate::storage::is_persistent_storage_error(source) + } + Self::ReplayUserOp { source } | Self::ReplayDirectInput { source } => { + app_error_is_terminal(source) + } + Self::NoSnapshot + | Self::SnapshotExecutionCountMismatch { .. } + | Self::ExecutionOffsetMismatch { .. } => true, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn app_internal() -> AppError { + AppError::Internal { + reason: "application invariant failed".into(), + } + } + + fn app_io() -> AppError { + AppError::Io(std::io::Error::other("filesystem unavailable")) + } + + fn app_io_kind(kind: std::io::ErrorKind) -> AppError { + AppError::Io(std::io::Error::from(kind)) + } + + #[test] + fn application_internal_errors_are_terminal_through_lane_wrappers() { + let errors = [ + InclusionLaneError::ExecuteUserOp { + source: app_internal(), + }, + InclusionLaneError::ExecuteDirectInput { + source: app_internal(), + }, + InclusionLaneError::LoadFromDump(app_internal()), + InclusionLaneError::CatchUp { + source: CatchUpError::ReplayUserOp { + source: app_internal(), + }, + }, + InclusionLaneError::CatchUp { + source: CatchUpError::ReplayDirectInput { + source: app_internal(), + }, + }, + InclusionLaneError::Snapshot(TakeDumpError::CreateDump(CreateDumpDirError::App( + app_internal(), + ))), + ]; + + for error in errors { + assert!(error.is_terminal_invariant(), "{error}"); + } + } + + #[test] + fn application_and_filesystem_io_errors_remain_operational() { + let errors = [ + InclusionLaneError::ExecuteUserOp { source: app_io() }, + InclusionLaneError::ExecuteDirectInput { source: app_io() }, + InclusionLaneError::LoadFromDump(app_io()), + InclusionLaneError::CatchUp { + source: CatchUpError::ReplayUserOp { source: app_io() }, + }, + InclusionLaneError::CatchUp { + source: CatchUpError::ReplayDirectInput { source: app_io() }, + }, + InclusionLaneError::Snapshot(TakeDumpError::CreateDump(CreateDumpDirError::App( + app_io(), + ))), + InclusionLaneError::Snapshot(TakeDumpError::CreateDump(CreateDumpDirError::Io( + std::io::Error::other("dump directory unavailable"), + ))), + InclusionLaneError::PromotionStamp(StampError::Io(std::io::Error::other( + "metadata unavailable", + ))), + ]; + + for error in errors { + assert!(!error.is_terminal_invariant(), "{error}"); + } + } + + #[test] + fn missing_or_corrupt_referenced_snapshot_is_terminal() { + for source in [ + app_io_kind(std::io::ErrorKind::NotFound), + app_io_kind(std::io::ErrorKind::InvalidData), + app_io_kind(std::io::ErrorKind::UnexpectedEof), + ] { + let error = InclusionLaneError::LoadFromDump(source); + assert!(error.is_terminal_invariant(), "{error}"); + } + } + + #[test] + fn persistent_snapshot_storage_errors_are_terminal() { + let errors = [ + InclusionLaneError::Snapshot(TakeDumpError::Storage( + rusqlite::Error::QueryReturnedNoRows, + )), + InclusionLaneError::Gc(GcError::Storage(rusqlite::Error::QueryReturnedNoRows)), + InclusionLaneError::PromotionStamp(StampError::Storage( + rusqlite::Error::QueryReturnedNoRows, + )), + ]; + + for error in errors { + assert!(error.is_terminal_invariant(), "{error}"); + } + } +} diff --git a/sequencer/src/ingress/inclusion_lane/mod.rs b/sequencer/src/ingress/inclusion_lane/mod.rs index 1d0766dc..40110b77 100644 --- a/sequencer/src/ingress/inclusion_lane/mod.rs +++ b/sequencer/src/ingress/inclusion_lane/mod.rs @@ -1,17 +1,26 @@ // (c) Cartesi and individual authors (see AUTHORS) // SPDX-License-Identifier: Apache-2.0 (see LICENSE) -//! Hot-path loop. The lane runs three layers of amortization on each iteration: +//! Single ordering lane with a latency-critical user-op regime and a slower L1 +//! reconciliation regime. It runs three layers of amortization: //! -//! - **Frontier check** (time-gated by `frontier_min_interval`): polls L1's -//! safe head; advances frame boundary if it moved. -//! - **Inner drain loop** (`run_inner_drain`): processes user-op chunks until -//! the queue empties or the batch hits its size target. -//! - **Per-chunk persistence** (`max_user_ops_per_chunk`): each chunk commits -//! in one SQL transaction, bounding ack latency for the first op in it. +//! - **Fast processing** (`run_fast_turn`): processes at most one bounded +//! user-op chunk per turn. Rejected requests therefore cannot keep a +//! continuously nonempty queue from starving reconciliation. +//! - **Per-chunk persistence** (`max_user_ops_per_chunk`): a nonempty accepted +//! subset commits at most once, bounding ack latency for its first op. +//! All-rejected chunks mutate nothing and do not open a transaction. +//! - **L1 reconciliation** (observed at `frontier_min_interval`): once five +//! newly-safe blocks have accumulated, consumes the complete range, promotes +//! snapshots, and advances one frame directly to the observed tip. The time +//! gate bounds SQL load; block distance is the semantic clock criterion. //! -//! The lane is a single-thread `spawn_blocking` task. SQLite is the only -//! synchronization with other components (input reader, batch submitter). +//! The lane is a single-thread `spawn_blocking` task. SQLite is the durable data +//! coordination boundary with the input reader and batch submitter. HTTP +//! ingress uses the deliberate bounded-channel request/response exception; +//! `RuntimeScope` is process control, not data coordination. Reconciliation +//! has no timeout/resume protocol: supported applications are assumed to +//! promptly digest the complete newly-safe range in the supported envelope. mod catch_up; mod config; @@ -25,6 +34,7 @@ mod tests; pub use config::InclusionLaneConfig; pub use error::InclusionLaneError; +pub(crate) use types::IncludedUserOp; pub use types::{PendingUserOp, SequencerError}; use std::thread; @@ -33,10 +43,10 @@ use std::time::{Duration, Instant, SystemTime}; use tokio::sync::mpsc; use tokio::task::JoinHandle; -use crate::runtime::shutdown::ShutdownSignal; -use crate::storage::{SafeInputRange, Storage, StoredSafeInput, WriteHead}; +use crate::runtime::shutdown::RuntimeScope; +use crate::storage::{SafeFrontierState, SafeInputRange, Storage, StoredSafeInput, WriteHead}; use sequencer_core::application::{ - AppError, Application, ExecutionOutcome, validate_and_execute_user_op, + AppError, Application, ExecutionOutcome, execute_direct_input, validate_and_execute_user_op, }; use sequencer_core::l2_tx::DirectInput; use sequencer_core::user_op::SignedUserOp; @@ -47,7 +57,7 @@ use catch_up::{catch_up_application, catch_up_snapshot}; /// receiver for the lifetime of the sequencer process. pub struct InclusionLane { rx: mpsc::Receiver, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, app: A, storage: Storage, config: InclusionLaneConfig, @@ -55,9 +65,9 @@ pub struct InclusionLane { impl InclusionLane { /// Spawn the lane on a blocking thread. The runtime establishes the open - /// Tip structurally before this — via [`Storage::ensure_open_tip`] (genesis) - /// or recovery's atomic reopen — so the lane only ever *loads* its resume - /// state and never initializes a Tip. It fail-louds with + /// Tip structurally before this — via the reducer's guarded + /// `EnsureOpenTip` phase or recovery's atomic reopen — so the lane only + /// ever *loads* its resume state and never initializes a Tip. It fail-louds with /// [`InclusionLaneError::NoOpenTip`] if the invariant was somehow violated. /// /// The lane selects one resume checkpoint — the latest pending @@ -73,9 +83,9 @@ impl InclusionLane { /// ops) and the join handle (for the runtime to observe lane /// shutdown). The handle resolves to `Ok(())` on graceful /// shutdown, or an `InclusionLaneError` if the lane crashed. - pub fn start( + pub(crate) fn start( queue_capacity: usize, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, storage: Storage, config: InclusionLaneConfig, ) -> ( @@ -93,6 +103,14 @@ impl InclusionLane { .map_err(|source| InclusionLaneError::CatchUp { source })?; let app = A::from_dump(&dump_info::app_prefix(&checkpoint.dump_dir)) .map_err(InclusionLaneError::LoadFromDump)?; + if app.executed_input_count() != checkpoint.executed_input_count { + return Err(InclusionLaneError::CatchUp { + source: error::CatchUpError::SnapshotExecutionCountMismatch { + application: app.executed_input_count().get(), + storage: checkpoint.executed_input_count.get(), + }, + }); + } tracing::debug!( l2_tx_index = checkpoint.l2_tx_index, "inclusion lane resuming from snapshot" @@ -113,8 +131,8 @@ impl InclusionLane { self.run_catch_up(catch_up_from)?; let mut included = Vec::with_capacity(self.config.max_user_ops_per_chunk.max(1)); let mut safe_inputs = Vec::with_capacity(self.config.safe_input_buffer_capacity.max(1)); - // The Tip exists by construction: the runtime established it via - // `Storage::ensure_open_tip` before the lane started. The lane only + // The Tip exists by construction: the startup reducer established it + // before runtime admission. The lane only // loads — read the open frame (fail-loud if absent) and the drain // cursor together from storage, so both come from the same place. Any // leading range already sequenced into the Tip's frames (genesis or a @@ -133,12 +151,21 @@ impl InclusionLane { return Ok(()); } + // Containment is consulted once per effect boundary, not per + // line: `run_fast_turn` checks on entry and before persist+ack, + // the batch-close branch below checks before its commit, and the + // reconciliation turn checks before its commit. Adjacent re-reads + // of the same bit buy a nanoseconds-narrower window in a design + // that already accepts the honest TOCTOU bound. self.maybe_advance_safe_frontier(&mut lane_state, &mut safe_inputs)?; - let drain = self.run_inner_drain(&mut lane_state.head, &mut included)?; + let turn = self.run_fast_turn(&mut lane_state.head, &mut included)?; - if drain.hit_batch_target() - || should_close_batch_by_time(&lane_state.head, &self.config) + if turn.hit_batch_target() || should_close_batch_by_time(&lane_state.head, &self.config) { + if self.shutdown.is_storage_invariant_contained() { + self.reject_pending_user_ops_due_to_shutdown(); + return Err(InclusionLaneError::TerminalStorageInvariant); + } let next_safe_block = lane_state.head.safe_block; // Atomic close: dump the app state, then seal the batch // and register its pending snapshot in one transaction. @@ -153,7 +180,7 @@ impl InclusionLane { &self.config.dumps_dir, ) .map_err(InclusionLaneError::Snapshot)?; - } else if !drain.drained_any() { + } else if !turn.processed_any() { // Nothing to drain and no batch to close: back off. GC no longer // lives here — it runs after a promotion in // `maybe_advance_safe_frontier`, so it tracks garbage creation @@ -173,38 +200,30 @@ impl InclusionLane { .map_err(|source| InclusionLaneError::CatchUp { source }) } - /// Drain user ops in chunks until the queue empties or we cross the batch - /// size target. Each chunk persists separately so ack latency stays bounded - /// by `max_user_ops_per_chunk`. - fn run_inner_drain( + /// Process at most one bounded dequeue chunk. Returning to the outer loop + /// does not imply an L1 query: the frontier check remains independently + /// time-gated, so fast turns normally run back-to-back. + fn run_fast_turn( &mut self, head: &mut WriteHead, - included: &mut Vec, - ) -> Result { - let mut drained_any = false; - loop { - let (count, outcome) = self.process_user_op_chunk(head, included)?; - if count > 0 { - drained_any = true; - } - match outcome { - ChunkOutcome::QueueEmpty => { - return Ok(if drained_any { - DrainSummary::DrainedQueue - } else { - DrainSummary::Idle - }); - } - ChunkOutcome::HitBatchTarget => return Ok(DrainSummary::HitBatchTarget), - ChunkOutcome::MoreToProcess => continue, - } + included: &mut Vec, + ) -> Result { + if self.shutdown.authorize().is_none() { + return Err(self.refuse_externalization(included)); + } + let (included_count, outcome) = self.process_user_op_chunk(head, included)?; + match outcome { + ChunkOutcome::HitBatchTarget => Ok(FastTurnSummary::HitBatchTarget), + ChunkOutcome::MoreToProcess => Ok(FastTurnSummary::Processed), + ChunkOutcome::QueueEmpty if included_count == 0 => Ok(FastTurnSummary::Idle), + ChunkOutcome::QueueEmpty => Ok(FastTurnSummary::Processed), } } fn process_user_op_chunk( &mut self, head: &mut WriteHead, - included: &mut Vec, + included: &mut Vec, ) -> Result<(usize, ChunkOutcome), InclusionLaneError> { included.clear(); let outcome = match dequeue_and_execute_user_op_chunk::( @@ -222,18 +241,22 @@ impl InclusionLane { }; let included_count = included.len(); - self.persist_included_user_ops(head, included)?; - - for item in included.drain(..) { - let _ = item.respond_to.send(Ok(())); - } + // Field-disjoint borrows: the token borrows `self.shutdown` while the + // commit mutably borrows `self.storage`; the acknowledgement function + // requires the token, so the FULL-committed-chunk-authorizes-ack + // boundary is a signature, not a convention. + let Some(auth) = self.shutdown.authorize() else { + return Err(refuse_externalization_parts(&mut self.rx, included)); + }; + persist_included_user_ops(&mut self.storage, head, included)?; + acknowledge_included(auth, included); Ok((included_count, outcome)) } - /// Time-gated to bound idle SQL load. High-throughput batches can delay - /// this past the gate, but a full batch is far less than 1s of work in - /// practice. + /// Time-gated to bound idle SQL load. The preceding fast turn is one + /// bounded dequeue chunk, so accepted and rejected traffic have the same + /// finite attempt bound before this method gets another opportunity. fn maybe_advance_safe_frontier( &mut self, lane_state: &mut LaneState, @@ -244,14 +267,34 @@ impl InclusionLane { } lane_state.mark_frontier_checked(); - let frontier = self.storage.safe_input_frontier()?; + let frontier = match self.storage.safe_frontier_state()? { + SafeFrontierState::Open(frontier) => frontier, + SafeFrontierState::CanonicalDivergence { + nonce, + safe_input_index, + } => { + self.reject_pending_user_ops_due_to_shutdown(); + return Err(InclusionLaneError::CanonicalDivergence { + nonce, + safe_input_index, + }); + } + }; assert!( frontier.end_exclusive >= lane_state.last_drained_direct_range.end(), "safe-input head regressed: safe_end={}, next={}", frontier.end_exclusive, lane_state.last_drained_direct_range.end() ); - if frontier.safe_block <= lane_state.head.safe_block { + assert!( + frontier.safe_block >= lane_state.head.safe_block, + "safe-block frontier regressed: observed={}, frame={}", + frontier.safe_block, + lane_state.head.safe_block, + ); + if frontier.safe_block - lane_state.head.safe_block + < sequencer_core::protocol::ProtocolTiming::FRAME_CLOCK_INTERVAL_SAFE_BLOCKS + { return Ok(()); } @@ -263,6 +306,10 @@ impl InclusionLane { // promoted-but-undrained batch — the state a restart would re-process // and re-promote on a deleted pending row. let observation = self.execute_safe_inputs_range(leading_direct_range, safe_inputs)?; + if self.shutdown.is_storage_invariant_contained() { + self.reject_pending_user_ops_due_to_shutdown(); + return Err(InclusionLaneError::TerminalStorageInvariant); + } let promoted = observation.commit( &mut self.storage, &mut lane_state.head, @@ -289,17 +336,10 @@ impl InclusionLane { Ok(()) } - fn persist_included_user_ops( - &mut self, - head: &mut WriteHead, - included: &mut Vec, - ) -> Result<(), InclusionLaneError> { - self.storage - .append_user_ops_chunk(head, included.as_slice()) - .map_err(|err| { - Self::respond_internal_to_all(included, "internal storage error".to_string()); - InclusionLaneError::Storage(err) - }) + /// Containment observed: refuse queued work and surface the terminal + /// class. The counterpart of a failed [`RuntimeScope::authorize`]. + fn refuse_externalization(&mut self, included: &mut Vec) -> InclusionLaneError { + refuse_externalization_parts(&mut self.rx, included) } /// Process the safe inputs in `direct_range`, accumulating which of our @@ -358,22 +398,27 @@ impl InclusionLane { payload: input.payload.clone(), }; - self.app - .execute_direct_input(&direct_input) + let receipt = execute_direct_input(&mut self.app, &direct_input) .map_err(|source| InclusionLaneError::ExecuteDirectInput { source })?; + observation.observe_direct_execution(crate::storage::DirectInputExecution { + safe_input_index, + executed_input_offset: receipt.offset, + }); } Ok(()) } - fn respond_internal_to_all(pending: &mut Vec, message: String) { + fn respond_internal_to_all(pending: &mut Vec, message: String) { for item in pending.drain(..) { let _ = item + .pending .respond_to .send(Err(SequencerError::internal(message.clone()))); } } fn reject_pending_user_ops_due_to_shutdown(&mut self) { + self.rx.close(); while let Ok(item) = self.rx.try_recv() { let _ = item .respond_to @@ -382,24 +427,74 @@ impl InclusionLane { } } +/// Commit the accepted chunk (`synchronous=FULL`). Only this commit +/// authorizes acknowledgements; a failed commit answers internal-error. +fn persist_included_user_ops( + storage: &mut Storage, + head: &mut WriteHead, + included: &mut Vec, +) -> Result<(), InclusionLaneError> { + storage + .append_executed_user_ops_chunk(head, included.as_slice()) + .map_err(|err| { + for item in included.drain(..) { + let _ = item.pending.respond_to.send(Err(SequencerError::internal( + "internal storage error".to_string(), + ))); + } + InclusionLaneError::Storage(err) + }) +} + +/// Acknowledge the FULL-committed chunk. Requires the externalization token: +/// a new acknowledgement site cannot skip the containment consult. +fn acknowledge_included( + _auth: crate::runtime::shutdown::Authorized<'_>, + included: &mut Vec, +) { + for item in included.drain(..) { + let _ = item.pending.respond_to.send(Ok(())); + } +} + +/// Shared refusal tail for a failed authorize: answer in-flight requests +/// unavailable, close intake, reject queued work, surface the terminal class. +fn refuse_externalization_parts( + rx: &mut mpsc::Receiver, + included: &mut Vec, +) -> InclusionLaneError { + for item in included.drain(..) { + let _ = item + .pending + .respond_to + .send(Err(SequencerError::unavailable("sequencer shutting down"))); + } + rx.close(); + while let Ok(item) = rx.try_recv() { + let _ = item + .respond_to + .send(Err(SequencerError::unavailable("sequencer shutting down"))); + } + InclusionLaneError::TerminalStorageInvariant +} + #[derive(Debug, PartialEq, Eq)] -enum DrainSummary { - /// Queue was empty; nothing was drained this pass. +enum FastTurnSummary { + /// The queue was observed empty and no accepted operation was persisted. Idle, - /// Drained the queue, no batch close needed (size-wise). - DrainedQueue, - /// Drained at least one op AND crossed the batch size target. - /// (`(false, true)` is unreachable: the size check fires only after a - /// successful execution, so `HitBatchTarget` always implies `drained_any`.) + /// Processed one chunk without crossing the batch target. This includes a + /// full all-rejected chunk, which must still yield to reconciliation. + Processed, + /// An accepted operation crossed the batch size target. HitBatchTarget, } -impl DrainSummary { +impl FastTurnSummary { fn hit_batch_target(&self) -> bool { matches!(self, Self::HitBatchTarget) } - fn drained_any(&self) -> bool { + fn processed_any(&self) -> bool { !matches!(self, Self::Idle) } } @@ -417,7 +512,7 @@ pub(super) enum ChunkOutcome { fn should_close_batch_by_time(head: &WriteHead, config: &InclusionLaneConfig) -> bool { // A backwards clock step makes `duration_since` err; `unwrap_or_default` // then reads as age 0, silently stalling the time-based close trigger - // until the clock catches up (review F8). Acceptable: the size trigger + // until the clock catches up. Acceptable: the size trigger // is unaffected, and a wedge here is liveness-only, never correctness. let age = SystemTime::now() .duration_since(head.batch_created_at) @@ -428,30 +523,30 @@ fn should_close_batch_by_time(head: &WriteHead, config: &InclusionLaneConfig) -> fn execute_user_op( app: &mut impl Application, item: PendingUserOp, - head: &WriteHead, - included: &mut Vec, + current_frame_fee: u16, + frame_safe_block: u64, + included: &mut Vec, ) -> Result<(), InclusionLaneError> { match validate_and_execute_user_op( app, item.signed.sender, &item.signed.user_op, - head.frame_fee, - head.safe_block, + current_frame_fee, + frame_safe_block, ) { - Ok(ExecutionOutcome::Included { .. }) => included.push(item), + Ok(ExecutionOutcome::Included(receipt)) => included.push(IncludedUserOp { + pending: item, + executed_input_offset: receipt.offset, + }), Ok(ExecutionOutcome::Invalid(reason)) => { let _ = item .respond_to .send(Err(SequencerError::invalid(reason.to_string()))); } - // Fail loud — the lane half of an asymmetry with the canonical fold - // (`execute_frame_user_ops` in `sequencer-core`), which silently *skips* - // this same `AppError`. Duality (I1) is preserved: an `AppError` excludes - // the op from state on both sides (here it is never pushed to `included`, - // there `outputs` is never extended), so the canonical state agrees. The - // lane additionally aborts because an error from a *validated* op is an - // internal-invariant breach, not a user-facing rejection — dead by - // construction today (see the matching note in the scheduler). + // Fail loud: an error from a validated op is an internal-invariant + // breach, not a user-facing rejection. The shared execution boundary + // does not advance scheduler-owned progress, and this op is never + // persisted or acknowledged. Err(err) => { let reason = match &err { AppError::Internal { reason } => reason.clone(), @@ -476,19 +571,22 @@ pub(super) fn dequeue_and_execute_user_op_chunk( app: &mut A, max_chunk: usize, head: &WriteHead, - included: &mut Vec, + included: &mut Vec, ) -> Result { let mut executed = 0_usize; while executed < max_chunk { match rx.try_recv() { Ok(item) => { - execute_user_op(app, item, head, included)?; - executed = executed.saturating_add(1); + execute_user_op(app, item, head.frame_fee, head.safe_block, included)?; + executed += 1; + let included_count = + u64::try_from(included.len()).expect("in-memory chunk length must fit in u64"); let projected = head .batch_user_op_count - .saturating_add(included.len() as u64); + .checked_add(included_count) + .expect("batch user-op count overflow: contract-impossible"); if user_op_count_to_bytes::(projected) >= head.max_batch_user_op_bytes { return Ok(ChunkOutcome::HitBatchTarget); } @@ -507,8 +605,15 @@ pub(super) fn dequeue_and_execute_user_op_chunk( } fn user_op_count_to_bytes(user_op_count: u64) -> u64 { - let one_user_op_bytes = SignedUserOp::max_batch_metadata() + A::MAX_METHOD_PAYLOAD_BYTES; - user_op_count.saturating_mul(one_user_op_bytes as u64) + let one_user_op_bytes = SignedUserOp::max_batch_metadata() + .checked_add(A::MAX_METHOD_PAYLOAD_BYTES) + .expect("one user-op wire bound overflow: contract-impossible"); + let one_user_op_bytes = + u64::try_from(one_user_op_bytes).expect("one user-op wire bound must fit in u64"); + // This is a comparison bound, not a persisted domain value: once the + // mathematical product exceeds u64::MAX it is certainly over every + // representable batch target, so clamping preserves the predicate exactly. + user_op_count.saturating_mul(one_user_op_bytes) } /// Lane-local state threaded through every loop iteration. diff --git a/sequencer/src/ingress/inclusion_lane/snapshot.rs b/sequencer/src/ingress/inclusion_lane/snapshot.rs index 8c219ced..2429ff87 100644 --- a/sequencer/src/ingress/inclusion_lane/snapshot.rs +++ b/sequencer/src/ingress/inclusion_lane/snapshot.rs @@ -16,17 +16,18 @@ //! batch of ours that landed in the range. At range close the lane //! promotes that one `(nonce, block)` target, folded into the same //! transaction that advances the drain -//! ([`crate::storage::Storage::close_frame_only_promoting`]) — so a -//! promotion and its drain commit atomically. Promotion is **per-range, -//! not per-block**: the range's max nonce supersedes every lower one, -//! and the skipped intermediate checkpoints were never observable. +//! ([`crate::storage::Storage::close_frame_only_promoting_with_executions`]) +//! — so promotion, drain, and canonical execution attributions commit +//! atomically. Promotion is **per-range, not per-block**: the range's max +//! nonce supersedes every lower one, and the skipped intermediate +//! checkpoints were never observable. //! //! 3. **After a promotion**, the lane runs [`run_gc`] to reclaim the //! now-superseded dump(s). GC tracks garbage creation, not idleness. //! -//! The observer's working state is one `Option<(nonce, block)>`; the lane -//! creates one per `execute_safe_inputs_range` call on the stack. No -//! allocation in the hot loop. +//! The observer holds one `Option<(nonce, block)>` plus direct-execution +//! receipts for the complete range. That allocation is confined to the slow +//! L1-reconciliation regime, not the user-op hot path. use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicU64, Ordering}; @@ -35,7 +36,7 @@ use std::time::{SystemTime, UNIX_EPOCH}; use sequencer_core::application::Application; use super::dump_info::{self, DumpInfo}; -use crate::storage::{SafeInputRange, Storage, WriteHead}; +use crate::storage::{DirectInputExecution, SafeInputRange, Storage, WriteHead}; /// Errors from snapshot-taking at batch close. #[derive(Debug, thiserror::Error)] @@ -146,6 +147,7 @@ pub(super) fn close_batch_with_snapshot( &dump_dir, nonce, l2_tx_index, + app.executed_input_count(), )?; Ok(()) } @@ -183,8 +185,8 @@ pub(super) fn take_dump_at_batch_close( /// block it landed in, then **commits itself once** at range close (via /// [`BlockObservation::commit`]): the promotion, if any, folds into the same /// transaction that advances the drain -/// ([`Storage::close_frame_only_promoting`]) — so a promotion and the drain it -/// derives from commit atomically. +/// ([`Storage::close_frame_only_promoting_with_executions`]) — so promotion, +/// drain, and canonical execution mappings commit atomically. /// /// Per-range (not per-block) promotion is sound because nonces land in /// monotonic order: the range's max nonce sits in its latest @@ -193,17 +195,23 @@ pub(super) fn take_dump_at_batch_close( /// observable anyway — `finalized` is a single async-polled row — so collapsing /// to one promotion loses nothing. /// -/// Constant memory: one `Option<(u64, u64)>`, no allocation, and infallible to -/// update (no storage on the hot path). +/// The accepted-batch observation is constant-sized. Direct execution receipts +/// are retained for the range so the eventual frame transaction can attach +/// their canonical offsets atomically; this allocation is confined to the +/// slow L1-reconciliation regime. pub(super) struct BlockObservation { /// `(nonce, inclusion_block)` of the highest accepted batch seen, or /// `None` if the range observed none of our batches. max: Option<(u64, u64)>, + direct_executions: Vec, } impl BlockObservation { pub(super) fn new() -> Self { - Self { max: None } + Self { + max: None, + direct_executions: Vec::new(), + } } /// Record a safe input belonging to L1 block `block`; if it was one of our @@ -218,9 +226,14 @@ impl BlockObservation { } } + pub(super) fn observe_direct_execution(&mut self, execution: DirectInputExecution) { + self.direct_executions.push(execution); + } + /// Close the frame for this safe-frontier advance, folding the observed /// promotion — if any — into the **same transaction** as the drain - /// ([`Storage::close_frame_only_promoting`]); otherwise a plain frame close. + /// ([`Storage::close_frame_only_promoting_with_executions`]); otherwise an + /// attributed plain frame close. /// Returns whether a batch was promoted, so the caller can collect the dumps /// it superseded. Consumes the observation — it is spent once committed. pub(super) fn commit( @@ -232,17 +245,23 @@ impl BlockObservation { ) -> Result { match self.max { Some((max_nonce, inclusion_block)) => { - storage.close_frame_only_promoting( + storage.close_frame_only_promoting_with_executions( head, next_safe_block, drained, + &self.direct_executions, max_nonce, inclusion_block, )?; Ok(true) } None => { - storage.close_frame_only(head, next_safe_block, drained)?; + storage.close_frame_only_with_executions( + head, + next_safe_block, + drained, + &self.direct_executions, + )?; Ok(false) } } @@ -277,7 +296,10 @@ mod tests { use std::sync::Mutex; use alloy_primitives::Address; - use sequencer_core::application::{AppError, AppOutputs, Application, InvalidReason}; + use sequencer_core::application::{ + AppError, AppOutputs, Application, ApplicationProgress, ApplyInputCapability, + InvalidReason, ProgressCommitCapability, + }; use sequencer_core::l2_tx::ValidUserOp; use sequencer_core::user_op::UserOp; @@ -291,12 +313,14 @@ mod tests { /// other trait methods aren't exercised in these tests. struct RecordingDumpApp { dumps: Mutex>, + progress: ApplicationProgress, } impl RecordingDumpApp { fn new() -> Self { Self { dumps: Mutex::new(Vec::new()), + progress: ApplicationProgress::default(), } } @@ -317,27 +341,32 @@ mod tests { Ok(()) } - fn execute_valid_user_op( + fn apply_valid_user_op( &mut self, + _capability: ApplyInputCapability<'_>, _user_op: &ValidUserOp, _safe_block: u64, ) -> Result { Ok(Vec::new()) } - fn execute_direct_input( + fn apply_direct_input( &mut self, + _capability: ApplyInputCapability<'_>, _input: &sequencer_core::l2_tx::DirectInput, ) -> Result { unimplemented!("not used in these tests") } - fn executed_input_count(&self) -> u64 { - 0 + fn execution_progress(&self) -> &ApplicationProgress { + &self.progress } - fn last_executed_safe_block(&self) -> u64 { - 0 + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + &mut self.progress } fn from_dump(_prefix: &Path) -> Result { @@ -484,7 +513,10 @@ mod tests { /// Application whose `create_dump` always fails — used to exercise /// the atomic-close failure path. - struct FailingDumpApp; + #[derive(Default)] + struct FailingDumpApp { + progress: ApplicationProgress, + } impl Application for FailingDumpApp { const MAX_METHOD_PAYLOAD_BYTES: usize = 0; @@ -498,31 +530,36 @@ mod tests { Ok(()) } - fn execute_valid_user_op( + fn apply_valid_user_op( &mut self, + _capability: ApplyInputCapability<'_>, _user_op: &ValidUserOp, _safe_block: u64, ) -> Result { Ok(Vec::new()) } - fn execute_direct_input( + fn apply_direct_input( &mut self, + _capability: ApplyInputCapability<'_>, _input: &sequencer_core::l2_tx::DirectInput, ) -> Result { unimplemented!("not used in these tests") } - fn executed_input_count(&self) -> u64 { - 0 + fn execution_progress(&self) -> &ApplicationProgress { + &self.progress } - fn last_executed_safe_block(&self) -> u64 { - 0 + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + &mut self.progress } fn from_dump(_prefix: &Path) -> Result { - Ok(FailingDumpApp) + Ok(Self::default()) } fn create_dump(&self, _prefix: &Path) -> Result<(), AppError> { @@ -550,7 +587,7 @@ mod tests { let open_before = head.batch_index; let dumps_dir = tempfile::tempdir().unwrap(); - let app = FailingDumpApp; + let app = FailingDumpApp::default(); let err = super::close_batch_with_snapshot(&app, &mut storage, &mut head, 0, dumps_dir.path()) .expect_err("create_dump failure must abort the close"); diff --git a/sequencer/src/ingress/inclusion_lane/tests.rs b/sequencer/src/ingress/inclusion_lane/tests.rs index bfbaa033..89eaf4d9 100644 --- a/sequencer/src/ingress/inclusion_lane/tests.rs +++ b/sequencer/src/ingress/inclusion_lane/tests.rs @@ -10,22 +10,56 @@ use app_core::application::MAX_METHOD_PAYLOAD_BYTES as WALLET_MAX_METHOD_PAYLOAD use rusqlite::params; use tokio::sync::{mpsc, oneshot}; -use crate::runtime::shutdown::ShutdownSignal; -use crate::storage::test_helpers::{SENDER_A, default_protocol_timing, temp_db}; -use crate::storage::{SafeInputRange, Storage, StoredSafeInput, WriteHead}; -use sequencer_core::application::{AppError, AppOutputs, Application, InvalidReason}; +use crate::runtime::shutdown::RuntimeScope; +use crate::storage::test_helpers::{ + SENDER_A, default_protocol_timing, pin_test_deployment_identity, record_canonical_divergence, + temp_db, +}; +use crate::storage::{DirectInputExecution, SafeInputRange, Storage, StoredSafeInput, WriteHead}; +use sequencer_core::application::{ + AppError, AppOutputs, Application, ApplicationProgress, ApplyInputCapability, InvalidReason, + ProgressCommitCapability, +}; +use sequencer_core::history::ExecutedInputCount; use sequencer_core::l2_tx::{DirectInput, SequencedL2Tx, ValidUserOp}; use sequencer_core::user_op::{SignedUserOp, UserOp}; -use super::catch_up::catch_up_application_paged; +use super::catch_up::{catch_up_application_paged, catch_up_snapshot}; use super::dequeue_and_execute_user_op_chunk; use super::error::CatchUpError; -use super::{InclusionLane, InclusionLaneConfig, InclusionLaneError, PendingUserOp}; +use super::{ + FastTurnSummary, IncludedUserOp, InclusionLane, InclusionLaneConfig, InclusionLaneError, + LaneState, PendingUserOp, SequencerError, +}; + +fn encode_progress(progress: ApplicationProgress) -> [u8; 16] { + let mut bytes = [0_u8; 16]; + bytes[..8].copy_from_slice(&progress.executed_input_count().get().to_le_bytes()); + bytes[8..].copy_from_slice(&progress.last_executed_safe_block().to_le_bytes()); + bytes +} + +fn decode_progress(bytes: &[u8], app_name: &str) -> Result { + if bytes.len() != 16 { + return Err(AppError::Internal { + reason: format!("{app_name} dump must be exactly 16 bytes"), + }); + } + let count = u64::from_le_bytes(bytes[..8].try_into().expect("checked slice length")); + let safe_block = u64::from_le_bytes(bytes[8..].try_into().expect("checked slice length")); + Ok(ApplicationProgress::new( + ExecutedInputCount::new(count), + safe_block, + )) +} #[derive(Default)] struct TestApp { nonces: HashMap, - executed_input_count: u64, + progress: ApplicationProgress, + /// Test-only scheduling seam used to keep a rejected queue saturated long + /// enough to distinguish one bounded turn from an unbounded drain. + reject_user_ops_after: Option, } impl Application for TestApp { @@ -34,35 +68,48 @@ impl Application for TestApp { fn validate_user_op( &self, _sender: Address, - _user_op: &UserOp, + user_op: &UserOp, _current_fee: u16, ) -> Result<(), InvalidReason> { + if let Some(delay) = self.reject_user_ops_after { + std::thread::sleep(delay); + return Err(InvalidReason::InvalidNonce { + expected: user_op.nonce.wrapping_add(1), + got: user_op.nonce, + }); + } Ok(()) } - fn execute_valid_user_op( + fn apply_valid_user_op( &mut self, + _capability: ApplyInputCapability<'_>, user_op: &ValidUserOp, _safe_block: u64, ) -> Result { let current = self.nonces.get(&user_op.sender).copied().unwrap_or(0); let next_nonce = current.wrapping_add(1); self.nonces.insert(user_op.sender, next_nonce); - self.executed_input_count = self.executed_input_count.saturating_add(1); Ok(Vec::new()) } - fn execute_direct_input(&mut self, _input: &DirectInput) -> Result { - self.executed_input_count = self.executed_input_count.saturating_add(1); + fn apply_direct_input( + &mut self, + _capability: ApplyInputCapability<'_>, + _input: &DirectInput, + ) -> Result { Ok(Vec::new()) } - fn executed_input_count(&self) -> u64 { - self.executed_input_count + fn execution_progress(&self) -> &ApplicationProgress { + &self.progress } - fn last_executed_safe_block(&self) -> u64 { - 0 + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + &mut self.progress } // The lane loads its app via `from_dump` after the runtime @@ -90,7 +137,10 @@ impl Application for TestApp { } } -struct InternalUserOpApp; +#[derive(Default)] +struct InternalUserOpApp { + progress: ApplicationProgress, +} impl Application for InternalUserOpApp { const MAX_METHOD_PAYLOAD_BYTES: usize = WALLET_MAX_METHOD_PAYLOAD_BYTES; @@ -104,8 +154,9 @@ impl Application for InternalUserOpApp { Ok(()) } - fn execute_valid_user_op( + fn apply_valid_user_op( &mut self, + _capability: ApplyInputCapability<'_>, _user_op: &ValidUserOp, _safe_block: u64, ) -> Result { @@ -114,20 +165,27 @@ impl Application for InternalUserOpApp { }) } - fn execute_direct_input(&mut self, _input: &DirectInput) -> Result { + fn apply_direct_input( + &mut self, + _capability: ApplyInputCapability<'_>, + _input: &DirectInput, + ) -> Result { unimplemented!("not used in these tests") } - fn executed_input_count(&self) -> u64 { - 0 + fn execution_progress(&self) -> &ApplicationProgress { + &self.progress } - fn last_executed_safe_block(&self) -> u64 { - 0 + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + &mut self.progress } fn from_dump(_prefix: &Path) -> Result { - Ok(InternalUserOpApp) + Ok(Self::default()) } fn create_dump(&self, prefix: &Path) -> Result<(), AppError> { @@ -160,7 +218,7 @@ enum ReplayEvent { } struct ReplayRecordingApp { - executed_input_count: u64, + progress: ApplicationProgress, replayed: Vec, } @@ -171,13 +229,13 @@ struct ReplayRecordingApp { /// its own instance via `from_dump`, so external observation has to /// go through the on-disk snapshot. struct SharedCountingApp { - executed_direct_inputs: u64, + progress: ApplicationProgress, } impl SharedCountingApp { fn new() -> Self { Self { - executed_direct_inputs: 0, + progress: ApplicationProgress::default(), } } } @@ -194,48 +252,45 @@ impl Application for SharedCountingApp { Ok(()) } - fn execute_valid_user_op( + fn apply_valid_user_op( &mut self, + _capability: ApplyInputCapability<'_>, _user_op: &ValidUserOp, _safe_block: u64, ) -> Result { Ok(Vec::new()) } - fn execute_direct_input(&mut self, _input: &DirectInput) -> Result { - self.executed_direct_inputs = self.executed_direct_inputs.saturating_add(1); + fn apply_direct_input( + &mut self, + _capability: ApplyInputCapability<'_>, + _input: &DirectInput, + ) -> Result { Ok(Vec::new()) } - fn executed_input_count(&self) -> u64 { - self.executed_direct_inputs + fn execution_progress(&self) -> &ApplicationProgress { + &self.progress } - fn last_executed_safe_block(&self) -> u64 { - 0 + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + &mut self.progress } fn from_dump(prefix: &Path) -> Result { let bytes = std::fs::read(Self::state_file_in_dump(prefix))?; - let counter = - u64::from_le_bytes( - bytes - .as_slice() - .try_into() - .map_err(|_| AppError::Internal { - reason: "SharedCountingApp dump must be exactly 8 bytes".to_string(), - })?, - ); - Ok(Self { - executed_direct_inputs: counter, - }) + let progress = decode_progress(bytes.as_slice(), "SharedCountingApp")?; + Ok(Self { progress }) } fn create_dump(&self, prefix: &Path) -> Result<(), AppError> { std::fs::create_dir(prefix)?; std::fs::write( Self::state_file_in_dump(prefix), - self.executed_direct_inputs.to_le_bytes(), + encode_progress(self.progress), )?; Ok(()) } @@ -253,7 +308,7 @@ impl Application for SharedCountingApp { impl ReplayRecordingApp { fn with_executed_input_count(executed_input_count: u64) -> Self { Self { - executed_input_count, + progress: ApplicationProgress::new(ExecutedInputCount::new(executed_input_count), 0), replayed: Vec::new(), } } @@ -277,8 +332,9 @@ impl Application for ReplayRecordingApp { Ok(()) } - fn execute_valid_user_op( + fn apply_valid_user_op( &mut self, + _capability: ApplyInputCapability<'_>, user_op: &ValidUserOp, _safe_block: u64, ) -> Result { @@ -286,26 +342,31 @@ impl Application for ReplayRecordingApp { sender: user_op.sender, data: user_op.data.clone(), }); - self.executed_input_count = self.executed_input_count.saturating_add(1); Ok(Vec::new()) } - fn execute_direct_input(&mut self, input: &DirectInput) -> Result { + fn apply_direct_input( + &mut self, + _capability: ApplyInputCapability<'_>, + input: &DirectInput, + ) -> Result { self.replayed.push(ReplayEvent::DirectInput { sender: input.sender, block_number: input.block_number, payload: input.payload.clone(), }); - self.executed_input_count = self.executed_input_count.saturating_add(1); Ok(Vec::new()) } - fn executed_input_count(&self) -> u64 { - self.executed_input_count + fn execution_progress(&self) -> &ApplicationProgress { + &self.progress } - fn last_executed_safe_block(&self) -> u64 { - 0 + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + &mut self.progress } fn from_dump(_prefix: &Path) -> Result { @@ -379,10 +440,11 @@ async fn start_lane( config: InclusionLaneConfig, ) -> ( mpsc::Sender, - ShutdownSignal, + RuntimeScope, tokio::task::JoinHandle>, ) { let mut storage = Storage::open(db_path).expect("open storage"); + pin_test_deployment_identity(&mut storage, config.batch_submitter_address); storage .append_safe_inputs(0, &[], SENDER_A, &default_protocol_timing()) .expect("seed observed safe head"); @@ -394,7 +456,7 @@ async fn start_lane( // Establish the tip structurally (as the runtime does at startup), then // hand its head to the lane — which now only loads, never initializes. storage.ensure_open_tip().expect("establish genesis tip"); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let (tx, handle) = InclusionLane::::start(128, shutdown.clone(), storage, config); (tx, shutdown, handle) } @@ -428,17 +490,454 @@ fn make_pending_user_op( ) } +fn make_included_user_op(seed: u8, offset: u64) -> IncludedUserOp { + let (pending, _response) = make_pending_user_op(seed); + IncludedUserOp { + pending, + executed_input_offset: ExecutedInputCount::new(offset), + } +} + +#[tokio::test] +async fn terminal_storage_fault_rejects_current_and_queued_ops_before_persistence() { + let db = temp_db("lane-terminal-storage-fault"); + let storage = Storage::open(db.path.as_str()).expect("open storage"); + let shutdown = RuntimeScope::default(); + let (tx, rx) = mpsc::channel(2); + let (current, current_response) = make_pending_user_op(0x61); + let (queued, queued_response) = make_pending_user_op(0x62); + tx.try_send(queued).expect("queue pending op"); + + let mut lane = InclusionLane { + rx, + shutdown: shutdown.clone(), + app: TestApp::default(), + storage, + config: default_test_config(), + }; + let mut included = vec![IncludedUserOp { + pending: current, + executed_input_offset: ExecutedInputCount::ZERO, + }]; + shutdown.contain_storage_invariant_failure("test fault"); + + assert!( + lane.shutdown.authorize().is_none(), + "containment must refuse the externalization token" + ); + assert!(matches!( + lane.refuse_externalization(&mut included), + InclusionLaneError::TerminalStorageInvariant + )); + assert!(included.is_empty()); + assert!(matches!( + current_response.await.expect("current response"), + Err(SequencerError::Unavailable(_)) + )); + assert!(matches!( + queued_response.await.expect("queued response"), + Err(SequencerError::Unavailable(_)) + )); + assert!( + lane.storage + .ordered_l2_txs_page_from(0, 1) + .expect("read ordered txs") + .is_empty() + ); +} + +#[test] +fn fast_turn_processes_at_most_one_rejected_chunk() { + let db = temp_db("one-rejected-chunk-per-fast-turn"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + storage + .append_safe_inputs(0, &[], SENDER_A, &default_protocol_timing()) + .expect("seed observed safe head"); + let mut head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + + let (tx, rx) = mpsc::channel(8); + let mut responses = Vec::new(); + for seed in 1..=8 { + let (mut pending, response) = make_pending_user_op(seed); + pending.signed.user_op.max_fee = 0; + tx.try_send(pending).expect("prefill rejected request"); + responses.push(response); + } + + let mut config = default_test_config(); + config.max_user_ops_per_chunk = 4; + let mut lane = InclusionLane { + rx, + shutdown: RuntimeScope::default(), + app: TestApp::default(), + storage, + config, + }; + let mut included = Vec::new(); + + let summary = lane + .run_fast_turn(&mut head, &mut included) + .expect("run one fast turn"); + + assert_eq!(summary, FastTurnSummary::Processed); + assert_eq!(lane.rx.len(), 4, "exactly one four-attempt chunk runs"); + assert_eq!(read_count(db.path.as_str(), "user_ops"), 0); + for response in responses.iter_mut().take(4) { + assert!(matches!( + response.try_recv(), + Ok(Err(SequencerError::Invalid(_))) + )); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn sustained_rejected_queue_cannot_starve_poisoned_frontier() { + let db = temp_db("reject-flood-does-not-starve-frontier"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + storage + .append_safe_inputs(0, &[], SENDER_A, &default_protocol_timing()) + .expect("seed observed safe head"); + storage.ensure_open_tip().expect("establish open tip"); + + let (tx, rx) = mpsc::channel(128); + let (first, first_response) = make_pending_user_op(0x80); + tx.try_send(first).expect("seed first rejected request"); + for seed in 1..128_u8 { + let (pending, _response) = make_pending_user_op(seed); + tx.try_send(pending).expect("saturate reject queue"); + } + + let shutdown = RuntimeScope::default(); + let mut config = default_test_config(); + config.max_user_ops_per_chunk = 4; + let mut lane = InclusionLane { + rx, + shutdown: shutdown.clone(), + app: TestApp { + reject_user_ops_after: Some(Duration::from_millis(2)), + ..TestApp::default() + }, + storage, + config, + }; + let mut lane_handle = tokio::task::spawn_blocking(move || lane.run_forever(0)); + + let producer_tx = tx.clone(); + let producer = tokio::spawn(async move { + let mut seed = 0_u8; + loop { + let (pending, _response) = make_pending_user_op(seed); + if producer_tx.send(pending).await.is_err() { + return; + } + seed = seed.wrapping_add(1); + } + }); + + let first_result = tokio::time::timeout(Duration::from_secs(1), first_response) + .await + .expect("lane must begin processing the saturated queue") + .expect("first response channel open"); + assert!(matches!(first_result, Err(SequencerError::Invalid(_)))); + + let mut poisoner = Storage::open(db.path.as_str()).expect("open divergence writer"); + record_canonical_divergence(&mut poisoner, 7, 0); + + let lane_result = match tokio::time::timeout(Duration::from_secs(1), &mut lane_handle).await { + Ok(joined) => joined.expect("join lane task"), + Err(_) => { + producer.abort(); + drop(tx); + shutdown.request_shutdown(); + let late_result = lane_handle.await.expect("join timed-out lane task"); + panic!("reject flood starved the poisoned frontier: {late_result:?}"); + } + }; + drop(tx); + producer.await.expect("join reject producer"); + + assert!(matches!( + lane_result, + Err(InclusionLaneError::CanonicalDivergence { + nonce: 7, + safe_input_index: 0, + }) + )); + assert_eq!(read_count(db.path.as_str(), "user_ops"), 0); +} + +#[test] +fn reconciliation_digests_an_epoch_sized_outage_backlog_in_one_turn() { + // The ADR's digestibility assumption — the complete accumulated + // newly-safe range is consumed in ONE reconciliation turn, with no + // timeout/resume protocol — exercised at L1-outage scale rather than + // the unit-sized ranges the clock tests use. Functional assertions only + // (everything drains, one frame at the observed tip); the printed wall + // time is developer evidence. The full ACK-latency-during-catch-up + // measurement stays with the benchmark harness. + const BACKLOG_DIRECTS: u64 = 5_000; + const JUMP_TARGET_BLOCK: u64 = 7_200; // ~a day of 12s safe blocks + + let db = temp_db("digestibility-epoch-backlog"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let config = default_test_config(); + pin_test_deployment_identity(&mut storage, config.batch_submitter_address); + storage + .append_safe_inputs(0, &[], SENDER_A, &default_protocol_timing()) + .expect("seed observed safe head"); + let head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + + let backlog: Vec = (0..BACKLOG_DIRECTS) + .map(|i| StoredSafeInput { + sender: Address::ZERO, + payload: vec![0xd1], + block_number: 1 + (i * (JUMP_TARGET_BLOCK - 1)) / BACKLOG_DIRECTS, + }) + .collect(); + storage + .append_safe_inputs( + JUMP_TARGET_BLOCK, + backlog.as_slice(), + SENDER_A, + &default_protocol_timing(), + ) + .expect("seed the outage backlog at the jump target"); + + let (_tx, rx) = mpsc::channel(1); + let mut lane = InclusionLane { + rx, + shutdown: RuntimeScope::default(), + app: TestApp::default(), + storage, + config, + }; + let mut state = LaneState::new(SafeInputRange::empty_at(0), head); + let mut safe_inputs = Vec::new(); + + let started = std::time::Instant::now(); + lane.maybe_advance_safe_frontier(&mut state, &mut safe_inputs) + .expect("one reconciliation turn digests the complete range"); + let elapsed = started.elapsed(); + + assert_eq!( + lane.storage.next_undrained_safe_input_index().unwrap(), + BACKLOG_DIRECTS, + "the complete accumulated range drains in one turn" + ); + assert_eq!( + read_frame_safe_blocks(db.path.as_str()), + vec![0, JUMP_TARGET_BLOCK], + "one frame at the observed tip; no synthetic intermediate ticks" + ); + println!( + "digestibility seed: {BACKLOG_DIRECTS} directs over a {JUMP_TARGET_BLOCK}-block jump \ + drained in one turn in {elapsed:?}" + ); +} + +#[test] +fn frame_clock_waits_five_blocks_and_collapses_observation_jumps() { + let db = temp_db("frame-clock-block-interval"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let config = default_test_config(); + pin_test_deployment_identity(&mut storage, config.batch_submitter_address); + storage + .append_safe_inputs(0, &[], SENDER_A, &default_protocol_timing()) + .expect("seed observed safe head"); + let head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + let (tx, rx) = mpsc::channel(1); + let mut lane = InclusionLane { + rx, + shutdown: RuntimeScope::default(), + app: TestApp::default(), + storage, + config, + }; + let mut state = LaneState::new(SafeInputRange::empty_at(0), head); + let mut safe_inputs = Vec::new(); + + lane.storage + .append_safe_inputs( + 4, + &[StoredSafeInput { + sender: Address::ZERO, + payload: vec![0xaa], + block_number: 4, + }], + SENDER_A, + &default_protocol_timing(), + ) + .expect("append below-threshold direct"); + lane.maybe_advance_safe_frontier(&mut state, &mut safe_inputs) + .expect("observe block 4"); + assert_eq!(read_frame_safe_blocks(db.path.as_str()), vec![0]); + assert_eq!(lane.storage.next_undrained_safe_input_index().unwrap(), 0); + + lane.storage + .append_safe_inputs(5, &[], SENDER_A, &default_protocol_timing()) + .expect("advance to first clock tick"); + lane.maybe_advance_safe_frontier(&mut state, &mut safe_inputs) + .expect("rotate at block 5"); + assert_eq!(read_frame_safe_blocks(db.path.as_str()), vec![0, 5]); + assert_eq!(read_frame_direct_count(db.path.as_str(), 0, 1), 1); + + let (pending, mut response) = make_pending_user_op(0x72); + tx.try_send(pending).expect("queue op after clock tick"); + let mut included = Vec::new(); + assert_eq!( + lane.run_fast_turn(&mut state.head, &mut included) + .expect("execute op at frame clock"), + FastTurnSummary::Processed + ); + assert!(matches!(response.try_recv(), Ok(Ok(())))); + let sequenced = lane + .storage + .ordered_l2_txs_page_from(0, 10) + .expect("read sequenced clock values"); + assert_eq!(sequenced.len(), 2); + assert!(matches!( + &sequenced[0].tx, + SequencedL2Tx::Direct(DirectInput { + block_number: 4, + .. + }) + )); + assert_eq!(sequenced[0].frame_safe_block, 5); + assert!(matches!(&sequenced[1].tx, SequencedL2Tx::UserOp(_))); + assert_eq!(sequenced[1].frame_safe_block, 5); + + lane.storage + .append_safe_inputs(32, &[], SENDER_A, &default_protocol_timing()) + .expect("jump safe head"); + lane.maybe_advance_safe_frontier(&mut state, &mut safe_inputs) + .expect("collapse jump to one frame"); + assert_eq!(read_frame_safe_blocks(db.path.as_str()), vec![0, 5, 32]); + + lane.storage + .append_safe_inputs(36, &[], SENDER_A, &default_protocol_timing()) + .expect("advance below reset threshold"); + lane.maybe_advance_safe_frontier(&mut state, &mut safe_inputs) + .expect("observe block 36"); + assert_eq!(read_frame_safe_blocks(db.path.as_str()), vec![0, 5, 32]); + + lane.storage + .append_safe_inputs(37, &[], SENDER_A, &default_protocol_timing()) + .expect("advance to reset threshold"); + lane.maybe_advance_safe_frontier(&mut state, &mut safe_inputs) + .expect("rotate at block 37"); + assert_eq!(read_frame_safe_blocks(db.path.as_str()), vec![0, 5, 32, 37]); + drop(tx); +} + +#[test] +fn structural_batch_frame_does_not_reset_frame_clock_anchor() { + let db = temp_db("batch-frame-does-not-reset-clock"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + storage + .append_safe_inputs(0, &[], SENDER_A, &default_protocol_timing()) + .expect("seed observed safe head"); + let head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + let (_tx, rx) = mpsc::channel(1); + let mut lane = InclusionLane { + rx, + shutdown: RuntimeScope::default(), + app: TestApp::default(), + storage, + config: default_test_config(), + }; + let mut state = LaneState::new(SafeInputRange::empty_at(0), head); + let mut safe_inputs = Vec::new(); + + lane.storage + .append_safe_inputs(4, &[], SENDER_A, &default_protocol_timing()) + .expect("advance below threshold"); + lane.maybe_advance_safe_frontier(&mut state, &mut safe_inputs) + .expect("observe block 4"); + let unchanged_clock = state.head.safe_block; + lane.storage + .close_frame_and_batch(&mut state.head, unchanged_clock) + .expect("create successor batch frame"); + assert_eq!(read_frame_safe_blocks(db.path.as_str()), vec![0, 0]); + + lane.storage + .append_safe_inputs(5, &[], SENDER_A, &default_protocol_timing()) + .expect("reach original five-block threshold"); + lane.maybe_advance_safe_frontier(&mut state, &mut safe_inputs) + .expect("clock tick after structural frame"); + assert_eq!(read_frame_safe_blocks(db.path.as_str()), vec![0, 0, 5]); +} + +#[test] +fn poisoned_frontier_outranks_frame_clock_and_closes_intake() { + let db = temp_db("poisoned-frontier-outranks-clock"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + storage + .append_safe_inputs(0, &[], SENDER_A, &default_protocol_timing()) + .expect("seed observed safe head"); + let head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + record_canonical_divergence(&mut storage, 7, 0); + + let (tx, rx) = mpsc::channel(2); + let (pending, mut response) = make_pending_user_op(0x70); + tx.try_send(pending) + .expect("queue request before poison read"); + let mut lane = InclusionLane { + rx, + shutdown: RuntimeScope::default(), + app: TestApp::default(), + storage, + config: default_test_config(), + }; + let mut state = LaneState::new(SafeInputRange::empty_at(0), head); + let mut safe_inputs = Vec::new(); + + let err = lane + .maybe_advance_safe_frontier(&mut state, &mut safe_inputs) + .expect_err("poison must prevent reconciliation even with zero block delta"); + + assert!(matches!( + &err, + InclusionLaneError::CanonicalDivergence { + nonce: 7, + safe_input_index: 0, + } + )); + assert!(err.is_terminal_invariant()); + assert!(matches!( + response.try_recv(), + Ok(Err(SequencerError::Unavailable(_))) + )); + let (late, _late_response) = make_pending_user_op(0x71); + assert!(matches!( + tx.try_send(late), + Err(mpsc::error::TrySendError::Closed(_)) + )); + assert_eq!(read_frame_safe_blocks(db.path.as_str()), vec![0]); +} + fn seed_replay_fixture(db_path: &str) -> Vec { let mut storage = Storage::open(db_path).expect("open storage"); + let batch_submitter_address = Address::from([0xff; 20]); + pin_test_deployment_identity(&mut storage, batch_submitter_address); let mut head = storage .initialize_open_state(0, SafeInputRange::empty_at(0)) .expect("initialize open state"); - let user_op_a = make_pending_user_op(0x51).0; - let user_op_b = make_pending_user_op(0x52).0; + let user_op_a = make_included_user_op(0x51, 0); + let user_op_b = make_included_user_op(0x52, 1); storage - .append_user_ops_chunk(&mut head, &[user_op_a, user_op_b]) - .expect("append first frame user ops"); + .append_executed_user_ops_chunk(&mut head, &[user_op_a, user_op_b]) + .expect("append attributed first-frame user ops"); storage .append_safe_inputs( 10, @@ -447,18 +946,26 @@ fn seed_replay_fixture(db_path: &str) -> Vec { payload: vec![0xaa], block_number: 10, }], - SENDER_A, + batch_submitter_address, &default_protocol_timing(), ) .expect("append first direct input"); storage - .close_frame_only(&mut head, 10, SafeInputRange::new(0, 1)) - .expect("close first frame"); + .close_frame_only_with_executions( + &mut head, + 10, + SafeInputRange::new(0, 1), + &[DirectInputExecution { + safe_input_index: 0, + executed_input_offset: ExecutedInputCount::new(2), + }], + ) + .expect("close first frame with direct attribution"); - let user_op_c = make_pending_user_op(0x53).0; + let user_op_c = make_included_user_op(0x53, 3); storage - .append_user_ops_chunk(&mut head, &[user_op_c]) - .expect("append second frame user op"); + .append_executed_user_ops_chunk(&mut head, &[user_op_c]) + .expect("append attributed second-frame user op"); storage .append_safe_inputs( 20, @@ -467,13 +974,21 @@ fn seed_replay_fixture(db_path: &str) -> Vec { payload: vec![0xbb], block_number: 20, }], - SENDER_A, + batch_submitter_address, &default_protocol_timing(), ) .expect("append second direct input"); storage - .close_frame_only(&mut head, 20, SafeInputRange::new(1, 2)) - .expect("close second frame"); + .close_frame_only_with_executions( + &mut head, + 20, + SafeInputRange::new(1, 2), + &[DirectInputExecution { + safe_input_index: 1, + executed_input_offset: ExecutedInputCount::new(4), + }], + ) + .expect("close second frame with direct attribution"); storage .append_safe_inputs( @@ -483,13 +998,21 @@ fn seed_replay_fixture(db_path: &str) -> Vec { payload: vec![0xcc], block_number: 30, }], - SENDER_A, + batch_submitter_address, &default_protocol_timing(), ) .expect("append third direct input"); storage - .close_frame_only(&mut head, 30, SafeInputRange::new(2, 3)) - .expect("close third frame"); + .close_frame_only_with_executions( + &mut head, + 30, + SafeInputRange::new(2, 3), + &[DirectInputExecution { + safe_input_index: 2, + executed_input_offset: ExecutedInputCount::new(5), + }], + ) + .expect("close third frame with direct attribution"); vec![ ReplayEvent::UserOp { @@ -542,6 +1065,21 @@ fn read_frame_direct_count(db_path: &str, batch_index: i64, frame_in_batch: i64) .expect("query frame direct count") } +fn read_frame_safe_blocks(db_path: &str) -> Vec { + let conn = Storage::open_connection(db_path).expect("open sqlite reader"); + let mut statement = conn + .prepare( + "SELECT safe_block FROM frames \ + ORDER BY batch_index ASC, frame_in_batch ASC", + ) + .expect("prepare frame-clock query"); + statement + .query_map([], |row| row.get::<_, i64>(0)) + .expect("query frame clocks") + .map(|value| u64::try_from(value.expect("read frame clock")).expect("nonnegative clock")) + .collect() +} + async fn wait_until(timeout: Duration, mut predicate: impl FnMut() -> bool) -> bool { let started = tokio::time::Instant::now(); while started.elapsed() < timeout { @@ -554,7 +1092,7 @@ async fn wait_until(timeout: Duration, mut predicate: impl FnMut() -> bool) -> b } async fn shutdown_lane( - shutdown: &ShutdownSignal, + shutdown: &RuntimeScope, handle: tokio::task::JoinHandle>, ) { shutdown.request_shutdown(); @@ -630,12 +1168,13 @@ async fn sequenced_safe_inputs_are_drained_but_not_executed() { batch_submitter_address, ..default_test_config() }; + pin_test_deployment_identity(&mut storage, batch_submitter_address); { let app = SharedCountingApp::new(); register_genesis_snapshot(&app, &mut storage, &config.dumps_dir); } storage.ensure_open_tip().expect("establish genesis tip"); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let (_tx, lane_handle) = InclusionLane::::start(128, shutdown.clone(), storage, config); let initialized = wait_until(Duration::from_secs(2), || { @@ -651,10 +1190,17 @@ async fn sequenced_safe_inputs_are_drained_but_not_executed() { 10, &[StoredSafeInput { sender: batch_submitter_address, - payload: vec![0xaa], + // A well-formed but unexpected-nonce batch is a production- + // faithful own-input row that the scheduler rejects. It must + // still advance the physical drain cursor without executing + // in the application. + payload: ssz::Encode::as_ssz_bytes(&sequencer_core::batch::Batch { + nonce: 1, + frames: Vec::new(), + }), block_number: 10, }], - SENDER_A, + batch_submitter_address, &default_protocol_timing(), ) .expect("append safe batch-submitter input"); @@ -670,18 +1216,30 @@ async fn sequenced_safe_inputs_are_drained_but_not_executed() { "expected sequenced safe input to be drained into frame 1" ); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let replay = storage + .ordered_l2_txs_page_from(0, 16) + .expect("load own-batch replay row"); + assert_eq!(replay.len(), 1); + assert!(matches!( + &replay[0].tx, + SequencedL2Tx::Direct(DirectInput { sender, .. }) + if *sender == batch_submitter_address + )); + assert_eq!(replay[0].executed_input_offset, None); + // The lane's own batch input was drained into a frame and // sequenced into `sequenced_l2_txs`, but the lane skipped - // `app.execute_direct_input` for it. Catch-up replays the same + // shared application-execution boundary for it. Catch-up replays the same // sequenced stream and also filters batch-submitter rows — so a // fresh `SharedCountingApp` driven through `catch_up_application` // ends with counter == 0, confirming the symmetric skip. - let mut storage = Storage::open(db.path.as_str()).expect("open storage"); let mut fresh_app = SharedCountingApp::new(); catch_up_application_paged(&mut fresh_app, &mut storage, batch_submitter_address, 0, 16) .expect("catch up"); assert_eq!( - fresh_app.executed_direct_inputs, 0, + fresh_app.executed_input_count().get(), + 0, "batch-submitter safe input should be skipped by the local app" ); } @@ -758,7 +1316,7 @@ async fn safe_inputs_already_available_are_sequenced_before_later_user_ops() { .ordered_l2_txs_page_from(0, 1_000_000) .expect("load ordered replay") .into_iter() - .map(|(_offset, tx, _frame_safe_block)| tx) + .map(|row| row.tx) .collect() }; shutdown_lane(&shutdown, lane_handle).await; @@ -901,7 +1459,7 @@ fn dequeue_returns_lane_error_when_app_reports_internal() { let (pending, recv) = make_pending_user_op(0x45); tx.blocking_send(pending).expect("enqueue pending user op"); - let mut app = InternalUserOpApp; + let mut app = InternalUserOpApp::default(); let mut included = Vec::new(); let head = unbounded_head(); let err = dequeue_and_execute_user_op_chunk(&mut rx, &mut app, 16, &head, &mut included) @@ -927,27 +1485,89 @@ fn catch_up_replays_multiple_pages() { let db = temp_db("catch-up-multi-page"); let expected = seed_replay_fixture(db.path.as_str()); let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let mappings: Vec<_> = storage + .ordered_l2_txs_page_from(0, 16) + .expect("load attributed replay rows") + .into_iter() + .map(|row| row.executed_input_offset.map(ExecutedInputCount::get)) + .collect(); + assert_eq!( + mappings, + vec![Some(0), Some(1), Some(2), Some(3), Some(4), Some(5)] + ); let mut app = ReplayRecordingApp::default(); catch_up_application_paged(&mut app, &mut storage, Address::from([0xff; 20]), 0, 2) .expect("catch up in pages"); assert_eq!(app.replayed, expected); - assert_eq!(app.executed_input_count(), expected.len() as u64); + assert_eq!(app.executed_input_count().get(), expected.len() as u64); + assert_eq!( + app.last_executed_safe_block(), + 30, + "catch-up must advance scheduler-owned progress through the shared boundary" + ); } #[test] -fn catch_up_replays_from_storage_even_when_app_reports_executed_inputs() { - let db = temp_db("catch-up-offset"); - let expected = seed_replay_fixture(db.path.as_str()); +fn catch_up_rejects_missing_mapping_before_execution() { + let db = temp_db("catch-up-missing-mapping"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + pin_test_deployment_identity(&mut storage, Address::from([0xff; 20])); + let mut head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + let (unmapped, _response) = make_pending_user_op(0x51); + storage + .append_user_ops_chunk(&mut head, &[unmapped]) + .expect("seed intentionally unmapped physical user op"); + + let mut app = ReplayRecordingApp::default(); + let err = catch_up_application_paged(&mut app, &mut storage, Address::from([0xff; 20]), 0, 2) + .expect_err("missing canonical mapping must stop catch-up"); + + assert!(matches!( + &err, + CatchUpError::ExecutionOffsetMismatch { + db_offset: 1, + kind: "user op", + expected: Some(0), + stored: None, + } + )); + assert!( + app.replayed.is_empty(), + "mapping is checked before execution" + ); + assert_eq!(app.executed_input_count(), ExecutedInputCount::ZERO); + assert!(InclusionLaneError::CatchUp { source: err }.is_terminal_invariant()); +} + +#[test] +fn catch_up_rejects_wrong_mapping_before_execution() { + let db = temp_db("catch-up-wrong-mapping"); + seed_replay_fixture(db.path.as_str()); let mut storage = Storage::open(db.path.as_str()).expect("open storage"); let mut app = ReplayRecordingApp::with_executed_input_count(3); - catch_up_application_paged(&mut app, &mut storage, Address::from([0xff; 20]), 0, 2) - .expect("catch up from storage"); + let err = catch_up_application_paged(&mut app, &mut storage, Address::from([0xff; 20]), 0, 2) + .expect_err("mapping from a different application boundary must stop catch-up"); - assert_eq!(app.replayed, expected); - assert_eq!(app.executed_input_count(), 3 + expected.len() as u64); + assert!(matches!( + &err, + CatchUpError::ExecutionOffsetMismatch { + db_offset: 1, + kind: "user op", + expected: Some(3), + stored: Some(0), + } + )); + assert!( + app.replayed.is_empty(), + "mapping is checked before execution" + ); + assert_eq!(app.executed_input_count(), ExecutedInputCount::new(3)); + assert!(InclusionLaneError::CatchUp { source: err }.is_terminal_invariant()); } #[test] @@ -963,6 +1583,271 @@ fn catch_up_handles_mixed_user_ops_and_direct_inputs_across_page_boundary() { assert_eq!(app.replayed, expected); } +#[test] +fn standard_recovery_rebases_history_and_restart_on_surviving_checkpoint() { + let db = temp_db("standard-recovery-history-boundary"); + let dumps_dir = tempfile::tempdir().expect("create dump directory"); + let batch_submitter = Address::from([0xff; 20]); + let direct_sender = Address::repeat_byte(0x44); + let protocol = default_protocol_timing(); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + pin_test_deployment_identity(&mut storage, batch_submitter); + storage + .append_safe_inputs(0, &[], batch_submitter, &protocol) + .expect("seed genesis safe head"); + storage.ensure_open_tip().expect("open genesis Tip"); + let mut head = storage.open_state().unwrap().unwrap(); + let initial_history = storage.history_state().expect("initial history"); + let mut live_app = SharedCountingApp::new(); + + // Batch 0 is the surviving prefix. Its pending snapshot reflects exactly + // input 0 and becomes the recovery checkpoint once the batch lands. + let (prefix_op, _response) = make_pending_user_op(0x51); + let mut included = Vec::new(); + super::execute_user_op( + &mut live_app, + prefix_op, + head.frame_fee, + head.safe_block, + &mut included, + ) + .expect("execute prefix user op"); + assert_eq!(included[0].executed_input_offset, ExecutedInputCount::ZERO); + storage + .append_executed_user_ops_chunk(&mut head, &included) + .expect("persist prefix user op"); + super::snapshot::close_batch_with_snapshot( + &live_app, + &mut storage, + &mut head, + 0, + dumps_dir.path(), + ) + .expect("close prefix batch with snapshot"); + + let prefix_landing = crate::storage::test_helpers::local_batch_payload(&mut storage, 0); + let direct = DirectInput { + sender: direct_sender, + block_number: 10, + payload: vec![0xd1], + }; + storage + .append_safe_inputs( + 10, + &[ + StoredSafeInput { + sender: batch_submitter, + payload: prefix_landing, + block_number: 10, + }, + StoredSafeInput { + sender: direct.sender, + payload: direct.payload.clone(), + block_number: direct.block_number, + }, + ], + batch_submitter, + &protocol, + ) + .expect("observe prefix landing and direct input"); + let direct_receipt = sequencer_core::application::execute_direct_input(&mut live_app, &direct) + .expect("execute direct input in doomed suffix"); + assert_eq!(direct_receipt.offset, ExecutedInputCount::new(1)); + storage + .close_frame_only_promoting_with_executions( + &mut head, + 10, + SafeInputRange::new(0, 2), + &[DirectInputExecution { + safe_input_index: 1, + executed_input_offset: direct_receipt.offset, + }], + 0, + 10, + ) + .expect("drain direct and promote prefix snapshot"); + let finalized_before = storage + .finalized_dump() + .expect("read finalized prefix") + .expect("prefix snapshot promoted"); + assert_eq!(finalized_before.l2_tx_index, 1); + assert_eq!( + finalized_before.executed_input_count, + ExecutedInputCount::new(1) + ); + + // The direct and this user op are both beyond the finalized checkpoint. + // Closing batch 1 makes it the first non-gold recovery pivot. + let (doomed_op, _response) = make_pending_user_op(0x52); + included.clear(); + super::execute_user_op( + &mut live_app, + doomed_op, + head.frame_fee, + head.safe_block, + &mut included, + ) + .expect("execute doomed user op"); + assert_eq!( + included[0].executed_input_offset, + ExecutedInputCount::new(2) + ); + storage + .append_executed_user_ops_chunk(&mut head, &included) + .expect("persist doomed user op"); + super::snapshot::close_batch_with_snapshot( + &live_app, + &mut storage, + &mut head, + 10, + dumps_dir.path(), + ) + .expect("close doomed batch with snapshot"); + assert_eq!( + storage.next_executed_input_count().unwrap(), + ExecutedInputCount::new(3) + ); + + let before_recovery = storage + .ordered_l2_txs_page_from(0, 32) + .expect("read pre-recovery history"); + let old_direct_physical = before_recovery + .iter() + .find_map(|row| match &row.tx { + SequencedL2Tx::Direct(value) if value.sender == direct_sender => { + assert_eq!(row.executed_input_offset, Some(ExecutedInputCount::new(1))); + Some(row.db_offset) + } + _ => None, + }) + .expect("doomed direct row exists before recovery"); + + let invalidated = storage + .recover_post_flush_for_recovery(10, &protocol, crate::clock::unix_now_ms()) + .expect("standard recovery cascade"); + assert_eq!(invalidated, vec![1, 2]); + let recovered_history = storage.history_state().expect("recovered history"); + assert_eq!( + recovered_history.version.era_id, initial_history.version.era_id, + "standard recovery must stay in the same era" + ); + assert_eq!(recovered_history.version.recovery_generation.get(), 1); + assert_eq!( + storage.next_executed_input_count().unwrap(), + ExecutedInputCount::new(2), + "H rolls back the doomed user op while the direct is re-drained" + ); + assert_eq!( + storage.finalized_dump().unwrap().unwrap(), + finalized_before, + "the accepted prefix checkpoint must survive the cascade" + ); + assert!( + storage.latest_pending_dump().unwrap().is_none(), + "the doomed suffix checkpoint must not survive" + ); + + let after_recovery = storage + .ordered_l2_txs_page_from(0, 32) + .expect("read recovered history"); + let (new_direct_physical, new_direct_logical) = after_recovery + .iter() + .find_map(|row| match &row.tx { + SequencedL2Tx::Direct(value) if value.sender == direct_sender => { + Some((row.db_offset, row.executed_input_offset)) + } + _ => None, + }) + .expect("re-drained direct exists after recovery"); + assert!( + new_direct_physical > old_direct_physical, + "recovery must physically re-drain the invalidated direct" + ); + assert_eq!(new_direct_logical, Some(ExecutedInputCount::new(1))); + + // A restart loads the surviving count-1 checkpoint and catches up through + // the replacement recovery Tip. The invalidated rows are physical audit + // history only and cannot perturb application progress. + let checkpoint = catch_up_snapshot(&mut storage).expect("select surviving checkpoint"); + assert_eq!(checkpoint.l2_tx_index, finalized_before.l2_tx_index); + assert_eq!( + checkpoint.executed_input_count, + finalized_before.executed_input_count + ); + let mut restarted = + SharedCountingApp::from_dump(&super::dump_info::app_prefix(&checkpoint.dump_dir)) + .expect("load surviving application checkpoint"); + assert_eq!(restarted.executed_input_count(), ExecutedInputCount::new(1)); + catch_up_application_paged( + &mut restarted, + &mut storage, + batch_submitter, + checkpoint.l2_tx_index, + 2, + ) + .expect("catch up through recovery re-drain"); + assert_eq!(restarted.executed_input_count(), ExecutedInputCount::new(2)); + + // The replacement suffix reuses offset 2, rolling H forward without + // retaining the invalidated op that previously occupied that coordinate. + let mut recovery_head = storage.open_state().unwrap().unwrap(); + let (replacement_op, _response) = make_pending_user_op(0x53); + included.clear(); + super::execute_user_op( + &mut restarted, + replacement_op, + recovery_head.frame_fee, + recovery_head.safe_block, + &mut included, + ) + .expect("execute replacement user op"); + assert_eq!( + included[0].executed_input_offset, + ExecutedInputCount::new(2) + ); + storage + .append_executed_user_ops_chunk(&mut recovery_head, &included) + .expect("persist replacement suffix"); + assert_eq!( + storage.next_executed_input_count().unwrap(), + ExecutedInputCount::new(3) + ); + + let valid = storage + .ordered_l2_txs_page_from(0, 32) + .expect("read replacement history"); + let mappings: Vec<_> = valid + .iter() + .map(|row| row.executed_input_offset.map(ExecutedInputCount::get)) + .collect(); + assert_eq!(mappings, vec![Some(0), None, Some(1), Some(2)]); + let user_seeds: Vec<_> = valid + .iter() + .filter_map(|row| match &row.tx { + SequencedL2Tx::UserOp(value) => Some(value.data[0]), + SequencedL2Tx::Direct(_) => None, + }) + .collect(); + assert_eq!(user_seeds, vec![0x51, 0x53]); + + let mut restarted_again = + SharedCountingApp::from_dump(&super::dump_info::app_prefix(&checkpoint.dump_dir)) + .expect("reload surviving checkpoint"); + catch_up_application_paged( + &mut restarted_again, + &mut storage, + batch_submitter, + checkpoint.l2_tx_index, + 2, + ) + .expect("catch up through replacement suffix"); + assert_eq!( + restarted_again.executed_input_count(), + ExecutedInputCount::new(3) + ); + assert_eq!(restarted_again.last_executed_safe_block(), 10); +} + #[test] fn catch_up_load_error_reports_offset() { let db = temp_db("catch-up-load-error"); @@ -975,19 +1860,59 @@ fn catch_up_load_error_reports_offset() { assert!(matches!(err, CatchUpError::LoadReplay { offset: 0, .. })); } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn lane_refuses_snapshot_whose_application_count_disagrees_with_storage() { + let db = temp_db("snapshot-execution-count-mismatch"); + let config = default_test_config(); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + pin_test_deployment_identity(&mut storage, config.batch_submitter_address); + storage + .append_safe_inputs( + 0, + &[], + config.batch_submitter_address, + &default_protocol_timing(), + ) + .expect("seed observed safe head"); + + let app = SharedCountingApp { + progress: ApplicationProgress::new(ExecutedInputCount::new(1), 0), + }; + register_genesis_snapshot(&app, &mut storage, &config.dumps_dir); + storage.ensure_open_tip().expect("establish genesis tip"); + + let shutdown = RuntimeScope::default(); + let (_tx, handle) = InclusionLane::::start(1, shutdown, storage, config); + let err = handle + .await + .expect("join lane startup") + .expect_err("snapshot count mismatch must refuse lane startup"); + + assert!(matches!( + &err, + InclusionLaneError::CatchUp { + source: CatchUpError::SnapshotExecutionCountMismatch { + application: 1, + storage: 0, + } + } + )); + assert!(err.is_terminal_invariant()); +} + /// App that counts executed user ops and persists the count through -/// `create_dump` / `from_dump` (8 LE bytes at `prefix/state`). The +/// `create_dump` / `from_dump` (two LE `u64`s at `prefix/state`). The /// restart-resume regression test reads this count back from the /// snapshot the lane writes on its own thread — the only way to observe /// how much state the lane rebuilt across a restart. struct UserOpCounterApp { - executed_user_ops: u64, + progress: ApplicationProgress, } impl UserOpCounterApp { fn new() -> Self { Self { - executed_user_ops: 0, + progress: ApplicationProgress::default(), } } } @@ -1004,48 +1929,45 @@ impl Application for UserOpCounterApp { Ok(()) } - fn execute_valid_user_op( + fn apply_valid_user_op( &mut self, + _capability: ApplyInputCapability<'_>, _user_op: &ValidUserOp, _safe_block: u64, ) -> Result { - self.executed_user_ops = self.executed_user_ops.saturating_add(1); Ok(Vec::new()) } - fn execute_direct_input(&mut self, _input: &DirectInput) -> Result { + fn apply_direct_input( + &mut self, + _capability: ApplyInputCapability<'_>, + _input: &DirectInput, + ) -> Result { unimplemented!("not used in these tests") } - fn executed_input_count(&self) -> u64 { - self.executed_user_ops + fn execution_progress(&self) -> &ApplicationProgress { + &self.progress } - fn last_executed_safe_block(&self) -> u64 { - 0 + fn execution_progress_mut( + &mut self, + _capability: ProgressCommitCapability<'_>, + ) -> &mut ApplicationProgress { + &mut self.progress } fn from_dump(prefix: &Path) -> Result { let bytes = std::fs::read(Self::state_file_in_dump(prefix))?; - let count = - u64::from_le_bytes( - bytes - .as_slice() - .try_into() - .map_err(|_| AppError::Internal { - reason: "UserOpCounterApp dump must be exactly 8 bytes".to_string(), - })?, - ); - Ok(Self { - executed_user_ops: count, - }) + let progress = decode_progress(bytes.as_slice(), "UserOpCounterApp")?; + Ok(Self { progress }) } fn create_dump(&self, prefix: &Path) -> Result<(), AppError> { std::fs::create_dir(prefix)?; std::fs::write( Self::state_file_in_dump(prefix), - self.executed_user_ops.to_le_bytes(), + encode_progress(self.progress), )?; Ok(()) } @@ -1063,12 +1985,10 @@ impl Application for UserOpCounterApp { fn read_dump_counter(dump_dir: &Path) -> u64 { let state_file = UserOpCounterApp::state_file_in_dump(&super::dump_info::app_prefix(dump_dir)); let bytes = std::fs::read(state_file).expect("read dump state file"); - u64::from_le_bytes( - bytes - .as_slice() - .try_into() - .expect("dump state must be 8 bytes"), - ) + decode_progress(bytes.as_slice(), "UserOpCounterApp") + .expect("decode dump progress") + .executed_input_count() + .get() } /// Regression for the resume-checkpoint bug: the lane used to load its @@ -1108,7 +2028,7 @@ async fn restart_resumes_from_pending_checkpoint_without_skipping_txs() { let app = UserOpCounterApp::new(); register_genesis_snapshot(&app, &mut storage, &config1.dumps_dir); storage.ensure_open_tip().expect("establish genesis tip"); - let shutdown1 = ShutdownSignal::default(); + let shutdown1 = RuntimeScope::default(); let (tx1, handle1) = InclusionLane::::start(128, shutdown1.clone(), storage, config1.clone()); assert!( @@ -1167,7 +2087,7 @@ async fn restart_resumes_from_pending_checkpoint_without_skipping_txs() { let mut storage2 = Storage::open(db.path.as_str()).expect("reopen storage"); // Restart: the Tip already exists, so this loads its head (warm path). storage2.ensure_open_tip().expect("load existing tip"); - let shutdown2 = ShutdownSignal::default(); + let shutdown2 = RuntimeScope::default(); let (tx2, handle2) = InclusionLane::::start(128, shutdown2.clone(), storage2, config2); @@ -1283,12 +2203,19 @@ fn promotion_advances_drain_atomically_so_restart_cannot_re_promote() { }), block_number: 100, }; + // DeferUntilAnchorSet skips acceptance simulation: this test's subject is + // promote/drain atomicity, and a hand-built landing payload cannot + // content-match an unsealed local batch — running the content-identity + // check here would record a divergence marker and (correctly) freeze the + // batch tree via the I15 triggers. storage - .append_safe_inputs( + .append_safe_inputs_with_timestamp( + 100, 100, std::slice::from_ref(&batch0), SENDER_A, &default_protocol_timing(), + crate::storage::FrontierMode::DeferUntilAnchorSet, ) .expect("append our batch as a safe input"); diff --git a/sequencer/src/ingress/inclusion_lane/types.rs b/sequencer/src/ingress/inclusion_lane/types.rs index b113db00..51715ded 100644 --- a/sequencer/src/ingress/inclusion_lane/types.rs +++ b/sequencer/src/ingress/inclusion_lane/types.rs @@ -6,6 +6,7 @@ use std::time::SystemTime; +use sequencer_core::history::ExecutedInputCount; use sequencer_core::user_op::SignedUserOp; use thiserror::Error; use tokio::sync::oneshot; @@ -19,6 +20,16 @@ pub struct PendingUserOp { pub received_at: SystemTime, } +/// A user op whose application mutation has succeeded but whose SQLite chunk +/// transaction has not committed yet. Keeping the receipt offset attached to +/// the request prevents the lane from persisting an included op without its +/// canonical application-history coordinate. +#[derive(Debug)] +pub(crate) struct IncludedUserOp { + pub pending: PendingUserOp, + pub executed_input_offset: ExecutedInputCount, +} + /// Per-op outcome reported back to the API caller via the response channel. /// /// - `Invalid` — application rejected the op (nonce mismatch, fee too low, etc.); maps to HTTP 4xx. diff --git a/sequencer/tests/batch_submitter_integration.rs b/sequencer/src/integration_tests/batch_submitter.rs similarity index 93% rename from sequencer/tests/batch_submitter_integration.rs rename to sequencer/src/integration_tests/batch_submitter.rs index c2906b6d..575067e0 100644 --- a/sequencer/tests/batch_submitter_integration.rs +++ b/sequencer/src/integration_tests/batch_submitter.rs @@ -11,13 +11,12 @@ use async_trait::async_trait; use sequencer::l1::submitter::{BatchPoster, BatchPosterError, TxHash}; use sequencer::l1::submitter::{BatchSubmitter, BatchSubmitterConfig}; use sequencer::l1::watermark::WalletNonceWatermarkSink; -use sequencer::runtime::shutdown::ShutdownSignal; +use sequencer::runtime::shutdown::RuntimeScope; use sequencer::storage::{SafeInputRange, Storage}; use sequencer_core::batch::Batch; use sequencer_core::protocol::ProtocolTiming; -mod common; -use common::{TestDb, temp_db}; +use super::common::{TestDb, temp_db}; /// Minimal mock for integration tests. /// @@ -60,6 +59,7 @@ impl TestMock { impl BatchPoster for TestMock { async fn submit_batches( &self, + _auth: crate::runtime::shutdown::Authorized<'_>, payloads: Vec>, _watermark: &dyn WalletNonceWatermarkSink, ) -> Result, BatchPosterError> { @@ -109,7 +109,7 @@ impl BatchPoster for TestMock { } } -/// Mirrors what `run_preemptive_recovery` does in production: persist a real +/// Mirrors what the startup recovery reducer establishes in production: persist a real /// safe-head observation so `submitter_frontier` has a row to read. Without /// this, the submitter's first tick errors out on `current_safe_block_required` /// and the loop exits before submitting anything. @@ -172,11 +172,16 @@ async fn submitter_loop_submits_closed_batches_then_exits_on_shutdown() { seed_two_closed_batches(&path); let mock = TestMock::new(); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let config = BatchSubmitterConfig { idle_poll_interval_ms: 5000, }; - let submitter = BatchSubmitter::new(path, mock.clone(), config); + let submitter = BatchSubmitter::new( + path, + mock.clone(), + config, + crate::runtime::process_lock::ProcessLock::test(), + ); let handle = submitter .start(shutdown.clone()) .expect("start batch submitter"); @@ -226,13 +231,18 @@ async fn submitter_re_enters_immediately_after_productive_tick() { let mock = TestMock::new(); mock.set_submit_delay(Duration::from_millis(400)); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let config = BatchSubmitterConfig { // Ten seconds — anything above ~2s would be enough to fail if the // immediate-retry cadence regressed to always-sleep. idle_poll_interval_ms: 10_000, }; - let submitter = BatchSubmitter::new(path.clone(), mock.clone(), config); + let submitter = BatchSubmitter::new( + path.clone(), + mock.clone(), + config, + crate::runtime::process_lock::ProcessLock::test(), + ); let handle = submitter .start(shutdown.clone()) .expect("start batch submitter"); @@ -280,14 +290,19 @@ async fn submitter_recovers_from_transient_poster_error_without_exiting() { let mock = TestMock::new(); mock.fail_next_n_submits(1); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let config = BatchSubmitterConfig { // Short poll interval so the retry sleep completes well within the // test window. Still long enough that accidentally always-sleeping // would delay the single submission past the assertion. idle_poll_interval_ms: 50, }; - let submitter = BatchSubmitter::new(path.clone(), mock.clone(), config); + let submitter = BatchSubmitter::new( + path.clone(), + mock.clone(), + config, + crate::runtime::process_lock::ProcessLock::test(), + ); let handle = submitter .start(shutdown.clone()) .expect("start batch submitter"); diff --git a/sequencer/tests/chain_id_validation.rs b/sequencer/src/integration_tests/chain_id_validation.rs similarity index 71% rename from sequencer/tests/chain_id_validation.rs rename to sequencer/src/integration_tests/chain_id_validation.rs index 3871d4e0..150f84d7 100644 --- a/sequencer/tests/chain_id_validation.rs +++ b/sequencer/src/integration_tests/chain_id_validation.rs @@ -13,7 +13,7 @@ //! must run `setup`). Fires before any L1 contact. //! - **Wrong signing key** → `IdentityError::Mismatch { batch_submitter_address }` //! (the key's address must match the pinned submitter). Fires before L1. -//! - **Wrong-chain RPC** (review F6) → `ChainIdMismatch`: a reachable RPC +//! - **Wrong-chain RPC** → `ChainIdMismatch`: a reachable RPC //! whose `eth_chainId` differs from the pinned chain id is refused. Needs //! a live RPC, so it uses Anvil. //! - **Matching-chain RPC** positive control: a matching chain must NOT @@ -23,7 +23,9 @@ use std::time::Duration; use alloy_primitives::{Address, address}; use clap::Parser; -use sequencer::runtime::{BootstrapError, IdentityError, RunError}; +use sequencer::commands::{BootstrapError, CommandError, IdentityError}; +use sequencer::l1::reader::InputReaderError; +use sequencer::recovery::{RecoveryError, RecoveryFailure}; use sequencer::storage::{DeploymentIdentity, Storage}; use sequencer::{Cli, Command}; use tempfile::TempDir; @@ -70,10 +72,17 @@ fn run_config(data_dir: &str, eth_rpc_url: &str, key: &str) -> sequencer::RunCon } /// Seed a pinned deployment identity (chain id `chain_id`, submitter -/// `submitter`) and mark setup complete — the minimal DB state `run` expects -/// from a completed `setup`, without `setup`'s L1 discovery. +/// `submitter`), finalized-snapshot fact, and completed setup lifecycle — the +/// minimal local state needed to reach `run`'s initial L1 sync without doing +/// `setup`'s on-chain discovery. Uses the same typed controller path `setup` +/// itself records through (admission → facts → completion), so this seed +/// cannot drift from the lifecycle schema or encode a state the controller +/// would never write (previously raw SQL from outside the crate). fn seed_setup_complete(db_path: &str, chain_id: u64, submitter: Address) { - let mut storage = Storage::open(db_path).expect("open db for seed"); + use sequencer::storage::LifecycleCommand; + + let mut storage = Storage::initialize_for_command(db_path, LifecycleCommand::Setup) + .expect("initialize setup lifecycle"); storage .load_or_insert_deployment_identity(DeploymentIdentity { chain_id, @@ -84,12 +93,16 @@ fn seed_setup_complete(db_path: &str, chain_id: u64, submitter: Address) { fee_oracle: sequencer::storage::FeeOracleIdentity::Fixed { log_gas_price: 0 }, }) .expect("seed deployment identity"); - storage.mark_setup_complete().expect("mark setup complete"); + let snapshot_prefix = std::path::Path::new(db_path).with_file_name("seed-finalized"); + storage + .insert_initial_finalized_dump(&snapshot_prefix, 0, 0, 0, 0) + .expect("seed finalized snapshot fact"); + storage.complete_setup().expect("complete setup"); } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn run_refuses_when_setup_incomplete() { - // Fresh DB, no marker: `run` must refuse before touching L1. + // Fresh DB, no lifecycle: `run` must refuse before touching L1. let dir = TempDir::new().expect("tempdir"); let data_dir = dir.path().to_str().unwrap(); let config = run_config(data_dir, "http://127.0.0.1:1", ANVIL_KEY); @@ -99,12 +112,12 @@ async fn run_refuses_when_setup_incomplete() { sequencer::run::(config), ) .await - .expect("run() must return quickly without a setup-complete marker"); + .expect("run() must return quickly without a completed setup lifecycle"); assert!( matches!( result, - Err(RunError::Bootstrap(BootstrapError::SetupNotComplete)) + Err(CommandError::Bootstrap(BootstrapError::SetupNotComplete)) ), "expected SetupNotComplete, got: {result:?}" ); @@ -128,7 +141,7 @@ async fn run_refuses_on_submitter_key_mismatch() { .expect("run() must return quickly on key/identity mismatch"); match result { - Err(RunError::Bootstrap(BootstrapError::Identity(IdentityError::Mismatch { + Err(CommandError::Bootstrap(BootstrapError::Identity(IdentityError::Mismatch { fields, .. }))) => assert_eq!(fields, "batch_submitter_address"), @@ -139,7 +152,7 @@ async fn run_refuses_on_submitter_key_mismatch() { #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn run_refuses_on_wrong_chain_rpc() { // Pinned chain id 31337, but the (reachable) RPC reports a different - // chain id — review F6: a wrong-chain RPC after setup must be refused. + // chain id: a wrong-chain RPC after setup must be refused. require_anvil(); let anvil = alloy::node_bindings::Anvil::default().chain_id(99).spawn(); let dir = TempDir::new().expect("tempdir"); @@ -158,9 +171,17 @@ async fn run_refuses_on_wrong_chain_rpc() { .expect("run() must return quickly on chain-id mismatch"); match result { - Err(RunError::Bootstrap(BootstrapError::ChainIdMismatch { rpc, config })) => { - assert_eq!(rpc, 99); - assert_eq!(config, 31_337); + Err(CommandError::Bootstrap(BootstrapError::Recovery(RecoveryError::Refuse(failure)))) => { + match *failure { + RecoveryFailure::InputReader(InputReaderError::ChainIdMismatch { + rpc, + expected, + }) => { + assert_eq!(rpc, 99); + assert_eq!(expected, 31_337); + } + other => panic!("expected input-reader chain-id mismatch, got: {other:?}"), + } } other => panic!("expected ChainIdMismatch, got: {other:?}"), } @@ -169,11 +190,9 @@ async fn run_refuses_on_wrong_chain_rpc() { #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn run_accepts_matching_chain_rpc() { // Positive control: a matching chain id must NOT produce ChainIdMismatch. - // The DB has identity + marker but no genesis snapshot, so `run` passes - // the chain-id check and then refuses at the always-load gate with - // SetupNotComplete — a deterministic proof the chain-id guard let it - // through (the gate sits immediately after the chain-id check, before any - // recovery write). + // The DB has a finalized-snapshot fact whose artifact is deliberately + // absent. `run` must pass the initial reader chain-id check and reach + // task-free runtime preparation, where loading that artifact fails. require_anvil(); let anvil = alloy::node_bindings::Anvil::default().spawn(); // chain id 31337 let dir = TempDir::new().expect("tempdir"); @@ -187,27 +206,12 @@ async fn run_accepts_matching_chain_rpc() { sequencer::run::(config), ) .await - .expect("run() returns promptly: chain-id passes, then the no-snapshot gate fires"); + .expect("run() returns promptly: chain-id passes, then snapshot preparation fails"); - // Assert the chain-id check positively did NOT fail, independent of which - // gate fires next: if `seed_setup_complete` ever starts registering a - // genesis snapshot, the `SetupNotComplete` arm below would stop firing and - // silently stop witnessing "chain id passed" — this negative pins it. + // The later artifact failure proves startup crossed the initial sync + // boundary, which includes the reader's chain-id verification. assert!( - !matches!( - result, - Err(RunError::Bootstrap( - BootstrapError::ChainIdMismatch { .. } | BootstrapError::ChainIdRpc { .. } - )) - ), - "matching chain id must not produce a chain-id error, got: {result:?}" - ); - assert!( - matches!( - result, - Err(RunError::Bootstrap(BootstrapError::SetupNotComplete)) - ), - "matching chain id must pass the chain-id check and reach the \ - always-load gate (SetupNotComplete), got: {result:?}" + matches!(result, Err(CommandError::ReferencedSnapshotArtifact { .. })), + "matching chain id must reach snapshot preparation, got: {result:?}" ); } diff --git a/sequencer/tests/common/mod.rs b/sequencer/src/integration_tests/common.rs similarity index 66% rename from sequencer/tests/common/mod.rs rename to sequencer/src/integration_tests/common.rs index 45b9afa5..b1c5be53 100644 --- a/sequencer/tests/common/mod.rs +++ b/sequencer/src/integration_tests/common.rs @@ -1,11 +1,7 @@ // (c) Cartesi and individual authors (see AUTHORS) // SPDX-License-Identifier: Apache-2.0 (see LICENSE) -//! Shared fixtures for `sequencer/tests/*.rs` integration tests. -//! -//! Integration tests compile as separate crates and cannot reach the -//! `#[cfg(test)]` helpers inside `sequencer/src/`. This module keeps the same -//! `TestDb` shape so callers work identically on both sides. +//! Shared fixtures for crate-internal integration-style tests. use tempfile::TempDir; diff --git a/sequencer/tests/e2e_sequencer.rs b/sequencer/src/integration_tests/e2e_sequencer.rs similarity index 95% rename from sequencer/tests/e2e_sequencer.rs rename to sequencer/src/integration_tests/e2e_sequencer.rs index 7f663571..0d5535e1 100644 --- a/sequencer/tests/e2e_sequencer.rs +++ b/sequencer/src/integration_tests/e2e_sequencer.rs @@ -17,8 +17,8 @@ use sequencer::http::{self, ApiConfig}; use sequencer::ingress::inclusion_lane::{ InclusionLane, InclusionLaneConfig, InclusionLaneError, PendingUserOp, dump_info, }; -use sequencer::runtime::shutdown::ShutdownSignal; -use sequencer::storage::{SafeInputRange, Storage, StoredSafeInput}; +use sequencer::runtime::shutdown::RuntimeScope; +use sequencer::storage::{DeploymentIdentity, FeeOracleIdentity, Storage, StoredSafeInput}; use sequencer_core::api::{TxRequest, TxResponse, WsTxMessage}; use sequencer_core::application::Application; use sequencer_core::l2_tx::SequencedL2Tx; @@ -29,8 +29,9 @@ use tokio::sync::mpsc; use tokio_tungstenite::connect_async; use tokio_tungstenite::tungstenite::Message; -mod common; -use common::temp_db; +use super::common::temp_db; + +const TEST_BATCH_SUBMITTER: Address = Address::repeat_byte(0xff); // ── V1 regression: cross-boundary signature domain consistency ──────── // @@ -202,7 +203,7 @@ async fn e2e_submit_tx_ack_and_broadcast() { // Fund the sender so the user-op passes the balance check. bootstrap_open_frame_with_deposits(db.path.as_str(), &[(sender, U256::from(1_000_000_u64))]); - let Some(runtime) = start_full_server(db.path.as_str(), domain.clone()).await else { + let Some(mut runtime) = start_full_server(db.path.as_str(), domain.clone()).await else { return; }; @@ -241,6 +242,20 @@ async fn e2e_submit_tx_ack_and_broadcast() { .submit_tx_with_status(&request_body) .await .expect("submit tx"); + if status != 200 + && runtime + .lane_handle + .as_ref() + .is_some_and(tokio::task::JoinHandle::is_finished) + { + let lane = runtime + .lane_handle + .take() + .expect("finished lane handle") + .await + .expect("join failed lane"); + panic!("lane stopped before acknowledgement: {lane:?}; body={response_body}"); + } assert_eq!( status, 200, "submit tx should succeed: body={response_body}" @@ -1044,7 +1059,7 @@ async fn restart_replays_same_ordered_l2_tx_stream_from_db() { struct FullServerRuntime { addr: std::net::SocketAddr, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, server_task: Option, lane_handle: Option< tokio::task::JoinHandle>, @@ -1086,7 +1101,7 @@ async fn start_full_server_with_max_body( let addr = listener.local_addr().expect("read listener addr"); let mut storage = Storage::open(db_path).expect("open storage"); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let dumps_dir = tempfile::tempdir().expect("e2e dumps dir").keep(); // Always-load invariant: ensure a finalized snapshot exists @@ -1112,17 +1127,24 @@ async fn start_full_server_with_max_body( .expect("register genesis"); } - // Tip-existence invariant: open the genesis Tip if the DB is fresh - // (no-op on warm restart), mirroring the runtime's structural startup. - // The lane loads the head itself after catch-up. + // Tip-existence invariant: open the genesis Tip only after the genesis + // snapshot exists. Opening the Tip may attribute already-ingested direct + // inputs, so doing this first would give the fresh application dump the + // wrong executed-input count. storage.ensure_open_tip().expect("establish open tip"); + let head = storage + .open_state() + .expect("load open state") + .expect("open state exists"); + // Default log_gas_price=0 -> 0+296+20+419+621 = 1356. + assert_eq!(head.frame_fee, 1356); let (tx, lane_handle) = InclusionLane::::start( 128, shutdown.clone(), storage, InclusionLaneConfig { - batch_submitter_address: Address::from([0xff; 20]), + batch_submitter_address: TEST_BATCH_SUBMITTER, dumps_dir, max_user_ops_per_chunk: 32, safe_input_buffer_capacity: 32, @@ -1138,20 +1160,20 @@ async fn start_full_server_with_max_body( L2TxFeedConfig { idle_poll_interval: Duration::from_millis(2), page_size: 64, - batch_submitter_address: None, + // Sentinel submitter: the WS assertions here observe the + // unfiltered stream, so pass an address no fixture seeds. + ..L2TxFeedConfig::new(alloy_primitives::Address::repeat_byte(0x7f)) }, ); let server_task = http::start_on_listener( listener, tx, - domain, - MAX_METHOD_PAYLOAD_BYTES, shutdown.clone(), tx_feed, ApiConfig { max_body_bytes, - ..ApiConfig::default() + ..ApiConfig::new(domain, MAX_METHOD_PAYLOAD_BYTES) }, http::SnapshotState { db_path: db_path.to_string(), @@ -1188,26 +1210,26 @@ async fn start_api_only_server( let _storage = Storage::open(db_path).expect("open storage"); let (tx, rx) = mpsc::channel::(queue_capacity); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let tx_feed = L2TxFeed::new( db_path.to_string(), shutdown.clone(), L2TxFeedConfig { idle_poll_interval: Duration::from_millis(2), page_size: 64, - batch_submitter_address: None, + // Sentinel submitter: the WS assertions here observe the + // unfiltered stream, so pass an address no fixture seeds. + ..L2TxFeedConfig::new(alloy_primitives::Address::repeat_byte(0x7f)) }, ); let server_task = http::start_on_listener( listener, tx, - domain, - MAX_METHOD_PAYLOAD_BYTES, shutdown.clone(), tx_feed, ApiConfig { max_body_bytes, - ..ApiConfig::default() + ..ApiConfig::new(domain, MAX_METHOD_PAYLOAD_BYTES) }, http::SnapshotState { db_path: db_path.to_string(), @@ -1256,14 +1278,24 @@ fn bootstrap_open_frame(db_path: &str) { bootstrap_open_frame_with_deposits(db_path, &[]); } -/// Bootstrap open frame, optionally seeding ERC-20 deposits for the given senders. -/// Each sender receives `amount` tokens before the frame is opened. +/// Seed the safe L1 head and optional ERC-20 deposits before runtime startup. +/// Each sender receives `amount` tokens when startup opens the first frame. fn bootstrap_open_frame_with_deposits(db_path: &str, deposits: &[(Address, U256)]) { let mut storage = Storage::open(db_path).expect("open storage"); let config = WalletConfig::default(); + storage + .load_or_insert_deployment_identity(DeploymentIdentity { + chain_id: 1, + app_address: Address::repeat_byte(0x11), + input_box_address: Address::repeat_byte(0x22), + app_deployment_block: 0, + batch_submitter_address: TEST_BATCH_SUBMITTER, + fee_oracle: FeeOracleIdentity::Fixed { log_gas_price: 0 }, + }) + .expect("pin test deployment identity"); // Always record a safe-head observation: production callers are gated by - // `run_preemptive_recovery`, so storage paths like `safe_input_frontier` + // the startup recovery reducer, so storage paths like `safe_input_frontier` // assume a row exists. With no deposits we still write an empty advance // so the lane can start without `current_safe_block_required` failing. let safe_inputs: Vec = deposits @@ -1284,7 +1316,7 @@ fn bootstrap_open_frame_with_deposits(db_path: &str, deposits: &[(Address, U256) .append_safe_inputs( 1, &safe_inputs, - Address::ZERO, + TEST_BATCH_SUBMITTER, &sequencer_core::protocol::ProtocolTiming { max_wait_blocks: sequencer_core::MAX_WAIT_BLOCKS, preemptive_margin_blocks: 75, @@ -1293,14 +1325,6 @@ fn bootstrap_open_frame_with_deposits(db_path: &str, deposits: &[(Address, U256) }, ) .expect("seed safe head (and any deposits)"); - - let safe_input_count = deposits.len() as u64; - let leading_range = SafeInputRange::new(0, safe_input_count); - // Default log_gas_price=0 → 0+296+20+419+621 = 1356. - let head = storage - .initialize_open_state(1, leading_range) - .expect("initialize open state"); - assert_eq!(head.frame_fee, 1356); } /// Default max_fee for test fixtures: must exceed default log_recommended_fee. @@ -1352,7 +1376,7 @@ fn all_ordered_l2_txs(db_path: &str) -> Vec { .ordered_l2_txs_page_from(0, 1_000_000) .expect("load ordered l2 txs") .into_iter() - .map(|(_offset, tx, _frame_safe_block)| tx) + .map(|row| row.tx) .collect() } diff --git a/sequencer/src/integration_tests/mod.rs b/sequencer/src/integration_tests/mod.rs new file mode 100644 index 00000000..8319d5fd --- /dev/null +++ b/sequencer/src/integration_tests/mod.rs @@ -0,0 +1,9 @@ +//! Integration-style tests kept inside the crate so raw worker launch APIs +//! can remain crate-private. Production consumers enter through `run_main`. + +mod batch_submitter; +mod chain_id_validation; +mod common; +mod e2e_sequencer; +mod snapshot_endpoints; +mod ws_broadcaster; diff --git a/sequencer/tests/snapshot_endpoints.rs b/sequencer/src/integration_tests/snapshot_endpoints.rs similarity index 98% rename from sequencer/tests/snapshot_endpoints.rs rename to sequencer/src/integration_tests/snapshot_endpoints.rs index 23a70947..19e3ecf3 100644 --- a/sequencer/tests/snapshot_endpoints.rs +++ b/sequencer/src/integration_tests/snapshot_endpoints.rs @@ -20,13 +20,12 @@ use sequencer::egress::l2_tx_feed::{L2TxFeed, L2TxFeedConfig}; use sequencer::http::{self, ApiConfig}; use sequencer::ingress::inclusion_lane::PendingUserOp; use sequencer::ingress::inclusion_lane::dump_info; -use sequencer::runtime::shutdown::ShutdownSignal; +use sequencer::runtime::shutdown::RuntimeScope; use sequencer::storage::Storage; use sequencer_core::application::Application; use tokio::sync::mpsc; -mod common; -use common::temp_db; +use super::common::temp_db; fn dummy_domain() -> Eip712Domain { Eip712Domain { @@ -42,7 +41,7 @@ fn dummy_domain() -> Eip712Domain { struct TestServer { addr: SocketAddr, _rx: mpsc::Receiver, - _shutdown: ShutdownSignal, + _shutdown: RuntimeScope, _task: http::ApiServerTask, } @@ -63,20 +62,19 @@ async fn start_server(db_path: &str) -> Option { }; let addr = listener.local_addr().expect("listener addr"); let (tx_sender, _rx) = mpsc::channel::(1); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); + // Sentinel submitter: this fixture seeds no own-batch rows. let tx_feed = L2TxFeed::new( db_path.to_string(), shutdown.clone(), - L2TxFeedConfig::default(), + L2TxFeedConfig::new(alloy_primitives::Address::repeat_byte(0x7f)), ); let task = http::start_on_listener( listener, tx_sender, - dummy_domain(), - MAX_METHOD_PAYLOAD_BYTES, shutdown.clone(), tx_feed, - ApiConfig::default(), + ApiConfig::new(dummy_domain(), MAX_METHOD_PAYLOAD_BYTES), http::SnapshotState { db_path: db_path.to_string(), state_file_in_dump: |dump_dir| { diff --git a/sequencer/tests/ws_broadcaster.rs b/sequencer/src/integration_tests/ws_broadcaster.rs similarity index 96% rename from sequencer/tests/ws_broadcaster.rs rename to sequencer/src/integration_tests/ws_broadcaster.rs index 2bb2c91f..3d44ee9f 100644 --- a/sequencer/tests/ws_broadcaster.rs +++ b/sequencer/src/integration_tests/ws_broadcaster.rs @@ -11,7 +11,7 @@ use futures_util::{SinkExt, StreamExt}; use sequencer::egress::l2_tx_feed::{L2TxFeed, L2TxFeedConfig}; use sequencer::http::{self, ApiConfig, WS_CATCHUP_WINDOW_EXCEEDED_REASON}; use sequencer::ingress::inclusion_lane::{PendingUserOp, SequencerError}; -use sequencer::runtime::shutdown::ShutdownSignal; +use sequencer::runtime::shutdown::RuntimeScope; use sequencer::storage::{SafeInputRange, Storage, StoredSafeInput}; use sequencer_core::api::WsTxMessage; use sequencer_core::l2_tx::SequencedL2Tx; @@ -21,8 +21,7 @@ use tokio::sync::{mpsc, oneshot}; use tokio_tungstenite::connect_async; use tokio_tungstenite::tungstenite::Message; -mod common; -use common::temp_db; +use super::common::temp_db; #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn ws_subscribe_streams_ordered_txs_from_offset_zero() { @@ -407,7 +406,7 @@ fn append_drained_direct_input(db_path: &str, payload: Vec) { struct WsServerRuntime { addr: std::net::SocketAddr, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, server_task: Option, } @@ -448,33 +447,35 @@ async fn start_test_server_with_limits( let addr = listener.local_addr().expect("read listener addr"); let (tx_sender, _rx) = mpsc::channel::(1); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let tx_feed = L2TxFeed::new( db_path.to_string(), shutdown.clone(), L2TxFeedConfig { idle_poll_interval: Duration::from_millis(2), page_size: 64, - batch_submitter_address: None, + // Sentinel submitter: this fixture seeds no own-batch rows. + ..L2TxFeedConfig::new(alloy_primitives::Address::repeat_byte(0x7f)) }, ); let task = http::start_on_listener( listener, tx_sender, - Eip712Domain { - name: None, - version: None, - chain_id: None, - verifying_contract: None, - salt: None, - }, - MAX_METHOD_PAYLOAD_BYTES, shutdown.clone(), tx_feed, ApiConfig { ws_max_subscribers, ws_max_catchup_events, - ..ApiConfig::default() + ..ApiConfig::new( + Eip712Domain { + name: None, + version: None, + chain_id: None, + verifying_contract: None, + salt: None, + }, + MAX_METHOD_PAYLOAD_BYTES, + ) }, http::SnapshotState { db_path: db_path.to_string(), @@ -558,7 +559,7 @@ fn load_ordered_l2_txs_page(db_path: &str, from_offset: u64, limit: usize) -> Ve .ordered_l2_txs_page_from(from_offset, limit) .expect("load ordered l2 tx page") .into_iter() - .map(|(_offset, tx, _frame_safe_block)| tx) + .map(|row| row.tx) .collect() } diff --git a/sequencer/src/l1/fee_oracle/bootstrap.rs b/sequencer/src/l1/fee_oracle/bootstrap.rs new file mode 100644 index 00000000..42de1091 --- /dev/null +++ b/sequencer/src/l1/fee_oracle/bootstrap.rs @@ -0,0 +1,67 @@ +// (c) Cartesi and individual authors (see AUTHORS) +// SPDX-License-Identifier: Apache-2.0 (see LICENSE) + +//! Setup-time construction of the Uniswap fee oracle. Setup owns the only +//! synchronous connect-and-quote requirement: it validates the source before +//! pinning and persists a real first price before completing. `run` starts +//! entirely from that persisted price and quotes from its supervised worker. + +use alloy::providers::DynProvider; + +use super::uniswap::{UniswapConfig, UniswapV3PriceSource, bootstrap_price_source_error}; +use super::worker::{FeeOracle, FeeOracleError}; +use crate::runtime::process_lock::ProcessLock; + +/// Failure to reach the configured pool at connect time. +#[derive(Debug)] +pub(crate) enum UniswapConnectError { + /// Wrong pool/pair/chain or a bad RPC URL: deterministic; retrying the + /// same configuration re-fails identically. + Misconfig(String), + /// The source is unavailable right now; retrying may succeed. + Transient { message: String }, +} + +/// Connect to the configured pool and classify any failure. `setup` calls +/// this while it still owns all address-bearing configuration and before +/// anything is pinned, so a misconfigured pool never pins an identity. +pub(crate) async fn connect_uniswap( + rpc_url: &str, + allow_insecure_rpc: bool, + uniswap: UniswapConfig, +) -> Result<(DynProvider, UniswapV3PriceSource), UniswapConnectError> { + let provider = crate::l1::provider::create_provider(rpc_url, allow_insecure_rpc) + .map_err(UniswapConnectError::Misconfig)?; + match UniswapV3PriceSource::connect(provider.clone(), uniswap).await { + Ok(token) => Ok((provider, token)), + Err(error) => { + let (transient, message) = bootstrap_price_source_error(error); + Err(if transient { + UniswapConnectError::Transient { message } + } else { + UniswapConnectError::Misconfig(message) + }) + } + } +} + +/// `setup`'s persist step: construct the oracle on an already-validated +/// `(provider, token)` pair, retain the data-directory lock, and quote once +/// so the pinned deployment always has a real first price. Setup requires +/// L1, so nothing is tolerated here. +pub(crate) async fn persist_first_price( + db_path: String, + provider: DynProvider, + token: UniswapV3PriceSource, + process_lock: ProcessLock, +) -> Result<(), FeeOracleError> { + let oracle = FeeOracle::new( + db_path, + FeeOracle::DEFAULT_POLL_INTERVAL, + provider, + Box::new(token), + process_lock, + ); + oracle.refresh_once().await?; + Ok(()) +} diff --git a/sequencer/src/l1/fee_oracle/mod.rs b/sequencer/src/l1/fee_oracle/mod.rs index f2fc0c6a..822597bf 100644 --- a/sequencer/src/l1/fee_oracle/mod.rs +++ b/sequencer/src/l1/fee_oracle/mod.rs @@ -3,9 +3,11 @@ //! L1-native fee-token gas-price oracle. +mod bootstrap; pub mod math; pub mod uniswap; pub mod worker; +pub(crate) use bootstrap::{UniswapConnectError, connect_uniswap, persist_first_price}; pub use uniswap::{TokenPriceSource, UniswapConfig, UniswapV3PriceSource}; pub use worker::FeeOracle; diff --git a/sequencer/src/l1/fee_oracle/uniswap.rs b/sequencer/src/l1/fee_oracle/uniswap.rs index ea6eae5d..04169d20 100644 --- a/sequencer/src/l1/fee_oracle/uniswap.rs +++ b/sequencer/src/l1/fee_oracle/uniswap.rs @@ -115,10 +115,16 @@ impl UniswapV3PriceSource { } else { return Err(PriceSourceError::WrongTokenPair); }; + // Uniswap V3 pools canonically order token0/token1 by address. Enforce + // that setup validated a real pool with the same ordering runtime later + // derives without another RPC round-trip. + if weth_is_token0 != (config.weth < config.fee_token) { + return Err(PriceSourceError::WrongTokenPair); + } // Probe the exact `observe` call used for quotes. Uniswap's `OLD` - // revert means this TWAP window cannot yet be served (misconfig / - // immature pool). Transport and other RPC failures stay provider - // errors so a flaky gateway is not treated as a bad pool. + // revert means this TWAP window cannot currently be served. Transport + // and `OLD` are availability failures; the remaining validation errors + // are deterministic configuration failures. pool.observe(vec![config.twap_window_secs, 0]) .call() .await @@ -131,6 +137,18 @@ impl UniswapV3PriceSource { }) } + /// Construct the runtime source from setup-pinned, setup-validated + /// identity without touching L1. Uniswap V3's canonical address ordering + /// supplies the only fact `connect` discovered that is needed to quote; + /// the input reader independently verifies the pinned chain on contact. + pub(crate) fn from_setup_validated(provider: DynProvider, config: UniswapConfig) -> Self { + Self { + provider, + config, + weth_is_token0: config.weth < config.fee_token, + } + } + async fn mean_tick(&self) -> Result { let pool = IUniswapV3Pool::new(self.config.pool, &self.provider); let observed = pool @@ -200,12 +218,14 @@ fn classify_decoded_observe_revert( } } -/// Bootstrap mapping: provider/RPC failures may self-heal; pool/config -/// failures need an operator. -pub fn bootstrap_price_source_error(error: PriceSourceError) -> (bool, String) { +/// Setup-time mapping: provider/RPC failures and an unavailable observation +/// window may self-heal. Both still fail setup's hard first-read requirement; +/// only their restart classification differs from deterministic misconfig. +pub(super) fn bootstrap_price_source_error(error: PriceSourceError) -> (bool, String) { match error { PriceSourceError::Provider(message) => (true, message), - other => (false, other.to_string()), + error @ PriceSourceError::InsufficientObservations => (true, error.to_string()), + error => (false, error.to_string()), } } @@ -404,17 +424,34 @@ mod tests { } #[test] - fn bootstrap_maps_provider_as_transient() { + fn bootstrap_maps_source_availability_as_transient() { let (transient, _) = bootstrap_price_source_error(PriceSourceError::Provider("timeout".into())); assert!(transient); let (transient, _) = bootstrap_price_source_error(PriceSourceError::InsufficientObservations); - assert!(!transient); + assert!(transient); let (transient, _) = bootstrap_price_source_error(PriceSourceError::WrongTokenPair); assert!(!transient); } + #[test] + fn runtime_source_derives_canonical_token_order_without_rpc() { + let provider = crate::l1::provider::create_provider("http://127.0.0.1:1", false) + .expect("provider construction is local"); + let source = UniswapV3PriceSource::from_setup_validated( + provider, + UniswapConfig { + chain_id: 1, + weth: MAINNET_WETH, + fee_token: MAINNET_USDC, + pool: MAINNET_USDC_WETH_005_POOL, + twap_window_secs: UniswapConfig::DEFAULT_TWAP_WINDOW_SECS, + }, + ); + assert_eq!(source.weth_is_token0, MAINNET_WETH < MAINNET_USDC); + } + #[test] fn boundary_ticks_are_representable() { assert!(quote_x_per_weth_from_tick(887_272, true).is_ok()); diff --git a/sequencer/src/l1/fee_oracle/worker.rs b/sequencer/src/l1/fee_oracle/worker.rs index 06c0322e..5203ba45 100644 --- a/sequencer/src/l1/fee_oracle/worker.rs +++ b/sequencer/src/l1/fee_oracle/worker.rs @@ -12,12 +12,9 @@ use tracing::{debug, warn}; use crate::l1::eip1559::{Eip1559Fees, estimate_fees}; use crate::l1::fee_oracle::math::{MathError, compute_x_units_per_gas, encode_log_gas_price}; -use crate::l1::fee_oracle::uniswap::{ - PriceSourceError, TokenPriceSource, UniswapConfig, UniswapV3PriceSource, - bootstrap_price_source_error, -}; -use crate::runtime::clock::unix_now_ms; -use crate::runtime::shutdown::ShutdownSignal; +use crate::l1::fee_oracle::uniswap::{PriceSourceError, TokenPriceSource}; +use crate::runtime::process_lock::{ProcessLock, spawn_blocking_with_lock}; +use crate::runtime::shutdown::RuntimeScope; use crate::storage::{Storage, StorageOpenError}; #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -42,6 +39,39 @@ pub enum FeeOracleError { FatalMath(#[from] MathError), } +impl FeeOracleError { + /// Whether this error poisons the run rather than restarting. Named + /// arms, no wildcard: a new variant must classify itself here. + pub(crate) fn is_terminal_invariant(&self) -> bool { + match self { + Self::Storage(source) => crate::storage::is_persistent_storage_error(source), + Self::OpenStorage(source) => crate::storage::is_persistent_storage_open_error(source), + // The oracle's `Join` wraps its internal blocking storage task. + // During a live worker exit that task can only have panicked; + // ordinary shutdown cancels the enclosing refresh future instead. + Self::FatalMath(_) | Self::Misconfig(_) | Self::Join(_) => true, + // A transient quote/transport failure self-heals. + Self::Transient(_) => false, + } + } + + /// Whether the worker should absorb this SQLite failure and retry its next + /// refresh. Contention is expected to self-heal under SQLite's single-writer + /// coordination; every other storage error keeps its ordinary supervisor + /// classification. + fn is_sqlite_contention(&self) -> bool { + let source = match self { + Self::Storage(source) => source, + Self::OpenStorage(StorageOpenError::Sqlite(source)) => source, + _ => return false, + }; + matches!( + source.sqlite_error_code(), + Some(rusqlite::ffi::ErrorCode::DatabaseBusy | rusqlite::ffi::ErrorCode::DatabaseLocked) + ) + } +} + #[async_trait] trait GasFeeSource: Send + Sync { async fn estimate_gas_fees(&self) -> Result; @@ -54,60 +84,33 @@ impl GasFeeSource for DynProvider { } } -enum TokenHandle { - Connected(Box), - /// Re-runs `UniswapV3PriceSource::connect` on every quote. Used when boot - /// skipped a live connect because L1 was transiently unreachable. - Reconnecting { - provider: DynProvider, - config: UniswapConfig, - }, -} - pub struct FeeOracle { db_path: String, poll_interval: Duration, - /// Shared with L1 read-staleness: `l1_read_stale_after_secs * 1000`. - max_price_age_ms: u64, gas: Box, - token: TokenHandle, + token: Box, + /// Retains data-directory exclusivity inside detached-capable blocking DB + /// work if the async setup/runtime task awaiting it is cancelled. + /// Required at construction. + process_lock: ProcessLock, } impl FeeOracle { pub const DEFAULT_POLL_INTERVAL: Duration = Duration::from_secs(12); - pub fn new( + pub(crate) fn new( db_path: impl Into, poll_interval: Duration, - max_price_age_ms: u64, provider: DynProvider, token: Box, + process_lock: ProcessLock, ) -> Self { Self { db_path: db_path.into(), poll_interval, - max_price_age_ms, gas: Box::new(provider), - token: TokenHandle::Connected(token), - } - } - - /// Uniswap worker that reconnects on each quote. Prefer [`Self::new`] when - /// boot already validated the pool; use this when boot tolerated a - /// transient connect failure and is running on a persisted price. - pub fn reconnecting_uniswap( - db_path: impl Into, - poll_interval: Duration, - max_price_age_ms: u64, - provider: DynProvider, - config: UniswapConfig, - ) -> Self { - Self { - db_path: db_path.into(), - poll_interval, - max_price_age_ms, - gas: Box::new(provider.clone()), - token: TokenHandle::Reconnecting { provider, config }, + token, + process_lock, } } @@ -115,56 +118,48 @@ impl FeeOracle { fn new_with_sources( db_path: impl Into, poll_interval: Duration, - max_price_age_ms: u64, gas: Box, token: Box, ) -> Self { Self { db_path: db_path.into(), poll_interval, - max_price_age_ms, gas, - token: TokenHandle::Connected(token), - } - } - - async fn quote_x_per_weth(&self) -> Result { - match &self.token { - TokenHandle::Connected(token) => token.quote_x_per_weth().await.map_err(Into::into), - TokenHandle::Reconnecting { provider, config } => { - let token = UniswapV3PriceSource::connect(provider.clone(), *config).await?; - token.quote_x_per_weth().await.map_err(Into::into) - } + token, + process_lock: ProcessLock::test(), } } /// Refresh from L1 and stamp `log_gas_price_updated_at_ms` even when the - /// encoded exponent is unchanged — the timestamp is the freshness signal. - pub async fn refresh_once(&self) -> Result { + /// encoded exponent is unchanged — the timestamp records the observation. + pub(super) async fn refresh_once(&self) -> Result { let fees = self .gas .estimate_gas_fees() .await .map_err(FeeOracleError::Transient)?; - let quote = self.quote_x_per_weth().await?; + let quote = self.token.quote_x_per_weth().await?; let linear = compute_x_units_per_gas(fees.base_fee_per_gas, fees.max_priority_fee_per_gas, quote)?; let log_gas_price = encode_log_gas_price(linear)?; let db_path = self.db_path.clone(); - let refresh = - tokio::task::spawn_blocking(move || -> Result { + let process_lock = self.process_lock.clone(); + let refresh = spawn_blocking_with_lock( + process_lock, + move || -> Result { let mut storage = Storage::open_writer(&db_path)?; let changed = storage.log_gas_price()? != log_gas_price; - // Always stamp: successful quote renews the staleness clock. + // Always stamp so operators can observe the last successful quote. storage.set_log_gas_price(log_gas_price)?; Ok(RefreshResult { log_gas_price, changed, }) - }) - .await - .map_err(|err| FeeOracleError::Join(err.to_string()))??; + }, + ) + .await + .map_err(|err| FeeOracleError::Join(err.to_string()))??; if refresh.changed { tracing::info!( @@ -190,30 +185,14 @@ impl FeeOracle { Ok(refresh) } - /// Refuse when the persisted price is missing or older than `max_age_ms`. - pub fn ensure_persisted_price_fresh( - db_path: &str, - max_age_ms: u64, - ) -> Result<(), FeeOracleError> { - let storage = Storage::open_read_only(db_path)?; - let now = unix_now_ms(); - if storage.log_gas_price_is_stale(now, max_age_ms)? { - let age = storage.log_gas_price_age_ms(now)?; - return Err(FeeOracleError::Transient(format!( - "persisted fee oracle price stale: age_ms={age}, max_age_ms={max_age_ms}" - ))); - } - Ok(()) - } - - pub fn start( + pub(crate) fn start( self, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, ) -> tokio::task::JoinHandle> { tokio::spawn(async move { self.run_forever(shutdown).await }) } - async fn run_forever(self, shutdown: ShutdownSignal) -> Result<(), FeeOracleError> { + async fn run_forever(self, shutdown: RuntimeScope) -> Result<(), FeeOracleError> { loop { tokio::select! { biased; @@ -225,30 +204,17 @@ impl FeeOracle { } } Err(FeeOracleError::Transient(error)) => { - let db_path = self.db_path.clone(); - let max_age_ms = self.max_price_age_ms; - let (age_ms, stale) = tokio::task::spawn_blocking(move || { - let storage = Storage::open_read_only(&db_path)?; - let now = unix_now_ms(); - let age_ms = storage.log_gas_price_age_ms(now)?; - let stale = storage.log_gas_price_is_stale(now, max_age_ms)?; - Ok::<_, FeeOracleError>((age_ms, stale)) - }) - .await - .map_err(|err| FeeOracleError::Join(err.to_string()))??; - if stale { - return Err(FeeOracleError::Transient(format!( - "fee oracle price older than {max_age_ms}ms \ - (age {age_ms}ms) after transient failure: {error}" - ))); - } warn!( %error, - retained_age_ms = age_ms, - max_age_ms, "retaining last L1 fee-oracle price after transient failure" ); } + Err(error) if error.is_sqlite_contention() => { + warn!( + error = %error, + "retaining last L1 fee-oracle price after SQLite contention" + ); + } Err(error) => return Err(error), }, } @@ -263,11 +229,15 @@ impl FeeOracle { impl From for FeeOracleError { fn from(error: PriceSourceError) -> Self { - let (transient, message) = bootstrap_price_source_error(error); - if transient { - Self::Transient(message) - } else { - Self::Misconfig(message) + match error { + PriceSourceError::Provider(message) => Self::Transient(message), + // Setup proved this window usable. At runtime `OLD` means the + // current view cannot serve it; retain the persisted price and let + // the input reader's safe-head policy judge shared-endpoint health. + error @ PriceSourceError::InsufficientObservations => { + Self::Transient(error.to_string()) + } + error => Self::Misconfig(error.to_string()), } } } @@ -279,8 +249,6 @@ mod tests { use alloy_primitives::U256; use std::sync::Mutex; - const TEST_MAX_AGE_MS: u64 = 60 * 60 * 1000; - struct StaticGas(Eip1559Fees); #[async_trait] @@ -317,6 +285,24 @@ mod tests { } } + struct FailsOnceGas { + calls: Mutex, + ok: Eip1559Fees, + } + + #[async_trait] + impl GasFeeSource for FailsOnceGas { + async fn estimate_gas_fees(&self) -> Result { + let mut calls = self.calls.lock().expect("lock"); + *calls += 1; + if *calls == 1 { + Err("rpc unavailable".into()) + } else { + Ok(self.ok) + } + } + } + struct FailsAfterFirstToken { calls: Mutex, ok: U256, @@ -344,6 +330,18 @@ mod tests { } } + struct PendingToken(Mutex>>); + + #[async_trait] + impl TokenPriceSource for PendingToken { + async fn quote_x_per_weth(&self) -> Result { + if let Some(entered) = self.0.lock().expect("lock").take() { + let _ = entered.send(()); + } + std::future::pending().await + } + } + struct OverflowToken; #[async_trait] @@ -378,17 +376,10 @@ mod tests { fn oracle_with( path: &str, - max_age_ms: u64, gas: Box, token: Box, ) -> FeeOracle { - FeeOracle::new_with_sources( - path.to_owned(), - Duration::from_secs(1), - max_age_ms, - gas, - token, - ) + FeeOracle::new_with_sources(path.to_owned(), Duration::from_secs(1), gas, token) } #[tokio::test] @@ -398,7 +389,6 @@ mod tests { let expected_log = expected_log_price(); let oracle = oracle_with( &db.path, - TEST_MAX_AGE_MS, Box::new(StaticGas(sample_fees())), Box::new(StaticToken(sample_quote())), ); @@ -415,7 +405,7 @@ mod tests { .unwrap(); assert!(first_stamp > 0); - // Unchanged exponent still renews the freshness stamp. + // Unchanged exponent still records the latest observation. tokio::time::sleep(Duration::from_millis(2)).await; assert_eq!( oracle.refresh_once().await.unwrap(), @@ -436,7 +426,6 @@ mod tests { let expected_log = expected_log_price(); let oracle = oracle_with( &db.path, - TEST_MAX_AGE_MS, Box::new(FailsAfterFirstGas { calls: Mutex::new(0), ok: sample_fees(), @@ -460,7 +449,6 @@ mod tests { let expected_log = expected_log_price(); let oracle = oracle_with( &db.path, - TEST_MAX_AGE_MS, Box::new(StaticGas(sample_fees())), Box::new(FailsAfterFirstToken { calls: Mutex::new(0), @@ -486,7 +474,6 @@ mod tests { let oracle = oracle_with( &db.path, - TEST_MAX_AGE_MS, Box::new(StaticGas(Eip1559Fees { base_fee_per_gas: u128::MAX, max_priority_fee_per_gas: u128::MAX, @@ -504,20 +491,23 @@ mod tests { } #[tokio::test] - async fn run_forever_shuts_down_cleanly() { + async fn run_forever_cancels_an_in_flight_refresh_on_shutdown() { let db = temp_db("fee-oracle-shutdown"); initialize_db(&db.path); + let (entered_tx, entered_rx) = tokio::sync::oneshot::channel(); let oracle = FeeOracle::new_with_sources( db.path.clone(), Duration::from_millis(50), - TEST_MAX_AGE_MS, Box::new(StaticGas(sample_fees())), - Box::new(StaticToken(sample_quote())), + Box::new(PendingToken(Mutex::new(Some(entered_tx)))), ); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let handle = oracle.start(shutdown.clone()); - tokio::time::sleep(Duration::from_millis(20)).await; + tokio::time::timeout(Duration::from_secs(2), entered_rx) + .await + .expect("refresh should enter token quote") + .expect("quote entry signal"); shutdown.request_shutdown(); tokio::time::timeout(Duration::from_secs(2), handle) .await @@ -535,14 +525,13 @@ mod tests { let oracle = FeeOracle::new_with_sources( db.path.clone(), Duration::from_millis(40), - TEST_MAX_AGE_MS, Box::new(FailsAfterFirstGas { calls: Mutex::new(0), ok: sample_fees(), }), Box::new(StaticToken(sample_quote())), ); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let mut handle = oracle.start(shutdown.clone()); tokio::select! { @@ -563,51 +552,116 @@ mod tests { } #[tokio::test] - async fn run_forever_exits_when_persisted_price_is_stale() { - let db = temp_db("fee-oracle-stale-exit"); + async fn run_forever_retries_a_failed_first_refresh() { + let db = temp_db("fee-oracle-first-refresh-retry"); + initialize_db(&db.path); + let expected_log = expected_log_price(); + let oracle = FeeOracle::new_with_sources( + db.path.clone(), + Duration::from_millis(20), + Box::new(FailsOnceGas { + calls: Mutex::new(0), + ok: sample_fees(), + }), + Box::new(StaticToken(sample_quote())), + ); + let shutdown = RuntimeScope::default(); + let mut handle = oracle.start(shutdown.clone()); + + tokio::time::timeout(Duration::from_secs(2), async { + loop { + if Storage::open_read_only(&db.path) + .unwrap() + .log_gas_price() + .unwrap() + == expected_log + { + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("worker should recover after its first transient refresh failure"); + assert!(!handle.is_finished(), "retry must not stop the worker"); + + shutdown.request_shutdown(); + tokio::time::timeout(Duration::from_secs(2), &mut handle) + .await + .expect("fee oracle exits within timeout") + .expect("join") + .expect("fee oracle result"); + } + + #[tokio::test] + async fn run_forever_retains_an_old_price_across_transient_failures() { + let db = temp_db("fee-oracle-old-price-retained"); initialize_db(&db.path); { let mut storage = Storage::open_writer(&db.path).unwrap(); storage.set_log_gas_price(42).unwrap(); - // Force an ancient stamp so the next transient fails the bound. storage.set_log_gas_price_updated_at_ms_for_test(1).unwrap(); } let oracle = FeeOracle::new_with_sources( db.path.clone(), Duration::from_millis(20), - 100, // 100ms max age Box::new(StaticGas(sample_fees())), Box::new(AlwaysFailToken), ); - let shutdown = ShutdownSignal::default(); - let handle = oracle.start(shutdown); + let shutdown = RuntimeScope::default(); + let mut handle = oracle.start(shutdown.clone()); + + tokio::select! { + biased; + result = &mut handle => panic!("transient source failure stopped worker: {result:?}"), + _ = tokio::time::sleep(Duration::from_millis(120)) => {} + } + let storage = Storage::open_read_only(&db.path).unwrap(); + assert_eq!(storage.log_gas_price().unwrap(), 42); + assert_eq!(storage.log_gas_price_updated_at_ms().unwrap(), 1); + drop(storage); - let result = tokio::time::timeout(Duration::from_secs(2), handle) + shutdown.request_shutdown(); + tokio::time::timeout(Duration::from_secs(2), handle) .await .expect("fee oracle exits within timeout") - .expect("join"); - let err = result.expect_err("stale price must stop the worker"); - assert!(matches!(err, FeeOracleError::Transient(ref m) if m.contains("older than"))); + .expect("join") + .expect("fee oracle result"); } #[test] - fn ensure_persisted_price_fresh_rejects_never_written() { - let db = temp_db("fee-oracle-never-written"); - initialize_db(&db.path); - let err = FeeOracle::ensure_persisted_price_fresh(&db.path, TEST_MAX_AGE_MS) - .expect_err("default stamp 0 is stale"); - assert!(matches!(err, FeeOracleError::Transient(_))); + fn sqlite_busy_and_locked_are_the_only_absorbed_storage_errors() { + fn sqlite_failure(code: rusqlite::ffi::ErrorCode) -> rusqlite::Error { + rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code, + extended_code: code as i32, + }, + None, + ) + } + + assert!( + FeeOracleError::Storage(sqlite_failure(rusqlite::ffi::ErrorCode::DatabaseBusy)) + .is_sqlite_contention() + ); + assert!( + FeeOracleError::OpenStorage(StorageOpenError::Sqlite(sqlite_failure( + rusqlite::ffi::ErrorCode::DatabaseLocked + ))) + .is_sqlite_contention() + ); + assert!( + !FeeOracleError::Storage(rusqlite::Error::QueryReturnedNoRows).is_sqlite_contention() + ); + assert!(!FeeOracleError::Misconfig("wrong pair".into()).is_sqlite_contention()); } #[test] - fn ensure_persisted_price_fresh_accepts_recent_write() { - let db = temp_db("fee-oracle-fresh-write"); - initialize_db(&db.path); - Storage::open_writer(&db.path) - .unwrap() - .set_log_gas_price(7) - .unwrap(); - FeeOracle::ensure_persisted_price_fresh(&db.path, TEST_MAX_AGE_MS).unwrap(); + fn runtime_treats_an_unavailable_observation_window_as_transient() { + let error = FeeOracleError::from(PriceSourceError::InsufficientObservations); + assert!(matches!(&error, FeeOracleError::Transient(_))); + assert!(!error.is_terminal_invariant()); } } diff --git a/sequencer/src/l1/mod.rs b/sequencer/src/l1/mod.rs index 0f980d7d..d0f3cb17 100644 --- a/sequencer/src/l1/mod.rs +++ b/sequencer/src/l1/mod.rs @@ -12,3 +12,86 @@ pub mod provider; pub mod reader; pub mod submitter; pub mod watermark; + +/// The L1 bundle handed whole to the entry points that reach L1 with the +/// submitter key: the batch-submitter provider builder and the +/// preemptive-recovery flush. Built once in `run` from the pinned +/// [`DeploymentIdentity`](crate::storage::DeploymentIdentity) plus the +/// command config; the identity is carried verbatim, never re-copied field +/// by field, so every consumer reads the pinned values through one route. +/// +/// It is not a funnel: the per-component configs (`InputReaderConfig`, +/// `BatchPosterConfig`, `UniswapConfig`) are built directly from the same +/// two sources — `identity` carries deployment values, and `RunConfig` +/// carries the per-client tuning knobs (`long_block_range_error_codes`, +/// poll intervals, confirmation depth) that belong to each client, not to +/// the L1 endpoint. Homed here (not with the command configs): L1-domain +/// identity consumed by mechanisms below the command layer. +#[derive(Debug, Clone)] +pub struct L1Config { + /// The pinned deployment identity, verbatim from the DB. Carried whole so + /// keyed-write paths (e.g. the preemptive-recovery flush) can re-confirm + /// the RPC's chain id right before signing via + /// [`crate::l1::provider::create_verified_signer_provider`]. + pub identity: crate::storage::DeploymentIdentity, + pub eth_rpc_url: String, + pub batch_submitter_private_key: SubmitterKey, + /// Opt into plaintext (`http://`) RPC against a non-loopback host — a + /// trusted private network (Docker/K8s service, private-VPC IP). Off by + /// default: the provider layer refuses remote plaintext otherwise. See + /// [`crate::l1::provider`]. + pub allow_insecure_rpc: bool, +} + +/// Hex-encoded batch-submitter signing key. `Debug` redacts and no `Display` +/// exists, so the secret cannot reach logs through formatting; the raw hex +/// is reachable only via [`Self::expose_secret`], keeping every consumer of +/// the secret greppable. The key's *public* identity needs no accessor — it +/// is the pinned `identity.batch_submitter_address` beside it in +/// [`L1Config`], verified against this key at the command gate. +#[derive(Clone)] +pub struct SubmitterKey(String); + +impl SubmitterKey { + pub fn new(hex: String) -> Self { + Self(hex) + } + + /// The raw hex. Every caller handles secret material — keep the call + /// sites few and auditable. + pub fn expose_secret(&self) -> &str { + &self.0 + } +} + +impl std::fmt::Debug for SubmitterKey { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("SubmitterKey([redacted])") + } +} + +#[cfg(test)] +mod tests { + use super::SubmitterKey; + + #[test] + fn submitter_key_debug_never_prints_the_secret() { + let key = SubmitterKey::new("0xdeadbeef_sentinel".to_string()); + let debug = format!("{key:?}"); + assert!(!debug.contains("sentinel"), "got: {debug}"); + assert_eq!(debug, "SubmitterKey([redacted])"); + // The derived Debug of a carrier struct inherits the redaction. + #[derive(Debug)] + #[allow(dead_code)] + struct Carrier { + key: SubmitterKey, + } + let carried = format!( + "{:?}", + Carrier { + key: SubmitterKey::new("0xdeadbeef_sentinel".to_string()) + } + ); + assert!(!carried.contains("sentinel"), "got: {carried}"); + } +} diff --git a/sequencer/src/l1/reader.rs b/sequencer/src/l1/reader.rs index 5be64181..64d5c024 100644 --- a/sequencer/src/l1/reader.rs +++ b/sequencer/src/l1/reader.rs @@ -22,7 +22,8 @@ use tracing::{debug, info}; use crate::l1::partition::{ GetLogsError, decode_evm_advance_input, get_input_added_events_ordered, }; -use crate::runtime::shutdown::ShutdownSignal; +use crate::runtime::process_lock::{ProcessLock, spawn_blocking_with_lock}; +use crate::runtime::shutdown::RuntimeScope; use crate::storage::{FrontierMode, IngestedSafeInput, Storage, StorageOpenError}; use sequencer_core::protocol::ProtocolTiming; @@ -42,9 +43,9 @@ pub struct InputReaderConfig { /// verifies the provider actually serves this chain on its first successful /// contact — the RPC URL is an operator CLI/env arg that may be repointed /// across restarts (token rotation, provider swap), so `run` cannot assume - /// it still matches the pinned identity. This backstops the boot-time check - /// ([`crate::runtime::validate_rpc_chain_id`]), which is skipped when L1 is - /// unreachable at boot. + /// it still matches the pinned identity. The check lives on the reader so + /// both startup recovery and the long-lived worker verify identity before + /// ingesting from the first successful provider contact. pub expected_chain_id: u64, } @@ -55,9 +56,9 @@ pub enum InputReaderError { #[error("provider/transport: {0}")] Provider(String), /// The provider answered, but with data that contradicts itself or the - /// InputBox's own accounting: an index gap or count mismatch (the F5 - /// witnesses), missing log provenance, an undecodable or self-inconsistent - /// `EvmAdvance` payload. Retryable — a consistent node returns a coherent + /// InputBox's own accounting: an index gap or count mismatch, missing log + /// provenance, an undecodable or self-inconsistent `EvmAdvance` payload. + /// Retryable — a consistent node returns a coherent /// set on the next tick — but logged as its own alarm: a *persistent* one /// means the endpoint is broken or lying, not slow. #[error("inconsistent L1 response: {0}")] @@ -75,10 +76,35 @@ pub enum InputReaderError { OpenStorage(#[from] StorageOpenError), #[error(transparent)] Storage(#[from] rusqlite::Error), + #[error("storage task panicked while {operation}: persistent invariant failure")] + StorageTaskPanicked { operation: &'static str }, #[error("input reader join error: {0}")] Join(String), } +impl InputReaderError { + /// Whether this error poisons the run rather than restarting. Named + /// arms, no wildcard: a new variant must classify itself here. + pub(crate) fn is_terminal_invariant(&self) -> bool { + match self { + Self::Storage(source) => crate::storage::is_persistent_storage_error(source), + Self::OpenStorage(source) => crate::storage::is_persistent_storage_open_error(source), + // A wrong-chain RPC caught after boot is a persistent operator + // misconfiguration; an inner storage-task panic preserves its + // provenance. + Self::ChainIdMismatch { .. } | Self::StorageTaskPanicked { .. } => true, + // Transport/L1-consistency errors self-heal on a healthy + // endpoint; `Bootstrap` is a preflight-phase fact (a live-worker + // appearance restarts unclassified rather than poisoning); a + // non-panic inner-task join is shutdown-path cancellation. + Self::Provider(_) + | Self::InconsistentL1Response(_) + | Self::Bootstrap(_) + | Self::Join(_) => false, + } + } +} + impl InputReaderError { /// Transient conditions the run loop logs and retries; everything else /// exits the worker. Exhaustive on purpose — a new variant must be @@ -90,6 +116,7 @@ impl InputReaderError { | Self::Bootstrap(_) | Self::OpenStorage(_) | Self::Storage(_) + | Self::StorageTaskPanicked { .. } | Self::Join(_) => false, } } @@ -121,14 +148,19 @@ pub struct InputReader { /// Until then every `advance_once` re-attempts the check, so an unreachable /// L1 keeps retrying and the verification fires the moment L1 returns. chain_id_verified: bool, + /// Retains exclusive data-directory ownership inside nested blocking DB + /// jobs if the async command/worker awaiting them is cancelled. Required + /// at construction: a reader without ownership is unrepresentable. + process_lock: ProcessLock, } impl InputReader { - pub async fn new( + pub(crate) async fn new( db_path: impl Into, config: InputReaderConfig, batch_submitter: Address, timing: ProtocolTiming, + process_lock: ProcessLock, ) -> Result { let provider = crate::l1::provider::create_provider(&config.rpc_url, config.allow_insecure_rpc) @@ -169,16 +201,18 @@ impl InputReader { db_path.into(), batch_submitter, timing, + process_lock, )) } - pub fn from_parts( + pub(crate) fn from_parts( config: InputReaderConfig, input_box_address: Address, app_deployment_block: u64, db_path: String, batch_submitter: Address, timing: ProtocolTiming, + process_lock: ProcessLock, ) -> Self { Self { config, @@ -189,6 +223,7 @@ impl InputReader { timing, frontier_mode: FrontierMode::Populate, chain_id_verified: false, + process_lock, } } @@ -212,17 +247,32 @@ impl InputReader { /// passing it at start time (instead of construction time) keeps the /// construction phase pure and ensures the same instance can't accidentally /// be started under two different shutdown signals. - pub fn start( + #[cfg(test)] + pub(crate) fn start( self, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, ) -> Result>, StorageOpenError> { + self.preflight_storage()?; + Ok(self.start_preflighted(shutdown)) + } + + /// Validate the storage dependency without starting a task. Runtime + /// startup calls this for every worker before launching any of them. + pub(crate) fn preflight_storage(&self) -> Result<(), StorageOpenError> { let _ = Storage::open(self.db_path.as_str())?; - Ok(tokio::spawn( - async move { self.run_forever(shutdown).await }, - )) + Ok(()) } - pub async fn sync_to_current_safe_head(&mut self) -> Result<(), InputReaderError> { + /// Spawn after [`Self::preflight_storage`] succeeded. Infallible so the + /// runtime can launch all workers in one non-yielding ownership step. + pub(crate) fn start_preflighted( + self, + shutdown: RuntimeScope, + ) -> JoinHandle> { + tokio::spawn(async move { self.run_forever(shutdown).await }) + } + + pub(crate) async fn sync_to_current_safe_head(&mut self) -> Result<(), InputReaderError> { let provider = crate::l1::provider::create_provider( &self.config.rpc_url, self.config.allow_insecure_rpc, @@ -232,12 +282,9 @@ impl InputReader { } /// Top-level driver. Races the work loop against the shutdown signal. - /// - /// `biased;` polls the shutdown arm first on every wakeup so a concurrent - /// shutdown wins over an in-flight `run_loop` step. Without `biased`, - /// `select!` would pick randomly between two ready branches and could - /// process one more iteration before shutting down. - async fn run_forever(self, shutdown: ShutdownSignal) -> Result<(), InputReaderError> { + /// Nested blocking DB jobs retain their own process-lock clone, so prompt + /// cancellation cannot release data-directory exclusivity under them. + async fn run_forever(self, shutdown: RuntimeScope) -> Result<(), InputReaderError> { tokio::select! { biased; _ = shutdown.wait_for_shutdown() => Ok(()), @@ -246,8 +293,7 @@ impl InputReader { } /// Tick → sleep → tick. Provider errors are logged and retried; other - /// errors propagate. Shutdown is handled by the outer `run_forever` - /// select, so this loop has no shutdown concerns. + /// errors propagate. Shutdown is handled by the outer biased select. async fn run_loop(mut self) -> Result<(), InputReaderError> { let provider = crate::l1::provider::create_provider( &self.config.rpc_url, @@ -301,7 +347,7 @@ impl InputReader { }; } - // F5 completeness witness, fetched *before* the scan. Pinning the count + // The input-completeness witness, fetched *before* the scan. Pinning the count // to `current_safe_block` (not latest) keeps it consistent with the // scanned range and forces the serving node to actually have that // block's state. Fetching it up front (rather than after `get_logs`, @@ -353,7 +399,7 @@ impl InputReader { } })?; - // F5: the InputBox assigns every input a per-app, gap-free `index`. We + // The InputBox assigns every input a per-app, gap-free `index`. We // ingest every event for this app from genesis, so each input's on-chain // index must equal the dense local `safe_input_index` it is assigned // (= the running count of already-stored inputs). Collect the on-chain @@ -381,7 +427,14 @@ impl InputReader { // deposit `≤` the safe head would be persisted as complete and only // caught on the *next* input, after the lane may have stamped frames // past it (divergence). - let received_total = expected_start.saturating_add(batch.len() as u64); + let received_count = u64::try_from(batch.len()).map_err(|_| { + InputReaderError::Provider("InputAdded result count exceeds u64".to_string()) + })?; + let received_total = expected_start.checked_add(received_count).ok_or_else(|| { + InputReaderError::Provider( + "InputAdded result count overflows the safe-input index space".to_string(), + ) + })?; check_input_count_complete(current_safe_block, received_total, onchain_count)?; info!( @@ -397,10 +450,10 @@ impl InputReader { /// on the first successful contact. A transport failure is retryable /// (`Provider`) and leaves the flag unset, so the check re-fires on the next /// tick until L1 is reachable; a value mismatch is fatal (`ChainIdMismatch`) - /// and propagates out of the run loop / aborts a recovery sync. This is the - /// backstop for `run`'s boot-time check, which is skipped when L1 is - /// unreachable at boot — without it, an RPC reconnecting on the wrong chain - /// would ingest address-filtered foreign logs unnoticed. + /// and propagates out of the run loop / aborts a recovery sync. Keeping the + /// check on the reader itself also covers a warm boot whose initial refresh + /// tolerated an unreachable provider: a later reconnect to the wrong chain + /// cannot ingest address-filtered foreign logs unnoticed. async fn verify_chain_id(&mut self, provider: &impl Provider) -> Result<(), InputReaderError> { if self.chain_id_verified { return Ok(()); @@ -423,23 +476,25 @@ impl InputReader { #[cfg(test)] async fn current_safe_block(&self) -> Result, InputReaderError> { let db_path = self.db_path.clone(); - tokio::task::spawn_blocking(move || { + let process_lock = self.process_lock.clone(); + spawn_blocking_with_lock(process_lock, move || { let mut storage = Storage::open(&db_path)?; storage.current_safe_block().map_err(InputReaderError::from) }) .await - .map_err(|err| InputReaderError::Join(err.to_string()))? + .map_err(|err| map_storage_task_join(err, "loading the current safe block"))? } /// Both scan-cursor reads in one connection: the persisted safe head /// (`None` before the first observation) and the count of already-stored /// safe inputs (= the next local `safe_input_index`, the expected on-chain - /// index of the next ingested input — F5). Reading them together is safe: + /// index of the next ingested input). Reading them together is safe: /// this reader is the sole writer of both, and `advance_once` holds /// `&mut self` on a single-task loop. async fn scan_cursor(&self) -> Result<(Option, u64), InputReaderError> { let db_path = self.db_path.clone(); - tokio::task::spawn_blocking(move || { + let process_lock = self.process_lock.clone(); + spawn_blocking_with_lock(process_lock, move || { let mut storage = Storage::open(&db_path)?; Ok(( storage.current_safe_block()?, @@ -447,7 +502,7 @@ impl InputReader { )) }) .await - .map_err(|err| InputReaderError::Join(err.to_string()))? + .map_err(|err| map_storage_task_join(err, "loading the safe-input cursor"))? } async fn append_safe_inputs( @@ -459,7 +514,8 @@ impl InputReader { let batch_submitter = self.batch_submitter; let timing = self.timing; let frontier_mode = self.frontier_mode; - tokio::task::spawn_blocking(move || { + let process_lock = self.process_lock.clone(); + spawn_blocking_with_lock(process_lock, move || { let mut storage = Storage::open(&db_path)?; storage .append_ingested_safe_inputs_with_timestamp( @@ -473,7 +529,18 @@ impl InputReader { .map_err(InputReaderError::from) }) .await - .map_err(|err| InputReaderError::Join(err.to_string()))? + .map_err(|err| map_storage_task_join(err, "appending safe inputs"))? + } +} + +/// Deliberately per-worker, not shared with the snapshot endpoint's +/// `storage_task`: this worker carries a typed error to the supervisor +/// through its exit channel, while an HTTP handler must contain immediately. +fn map_storage_task_join(err: tokio::task::JoinError, operation: &'static str) -> InputReaderError { + if err.is_panic() { + InputReaderError::StorageTaskPanicked { operation } + } else { + InputReaderError::Join(err.to_string()) } } @@ -585,7 +652,7 @@ fn u256_to_u64(value: U256, what: &str) -> Result { } /// Decode one `InputAdded` log into the row we persist, plus the on-chain -/// index the F5 contiguity witness checks. Pure — unit tests drive it with +/// index the contiguity witness checks. Pure — unit tests drive it with /// synthetic `(InputAdded, Log)` pairs, no provider required. Mirrors the /// watchdog's `decode_and_validate_log`: the row mixes provenances (block /// number and tx hash from the log, everything else from the `EvmAdvance` @@ -644,7 +711,7 @@ fn ingest_input_added( /// every input a per-app, gap-free index, and we ingest every event for this app /// from genesis, so the indices must be `expected_start, expected_start+1, …`. A /// gap means the provider returned an incomplete `get_logs` set (a clamped or -/// lagging-replica response — F5, see `docs/threat-model/README.md`); the +/// lagging-replica response — see `docs/threat-model/README.md`); the /// retryable [`InconsistentL1Response`] refuses to persist the hole — a /// consistent provider returns the full set on the next tick. /// @@ -654,11 +721,18 @@ fn check_input_index_contiguity( onchain_indices: &[u64], ) -> Result<(), InputReaderError> { for (offset, &index) in onchain_indices.iter().enumerate() { - let expected = expected_start.saturating_add(offset as u64); + let offset = u64::try_from(offset).map_err(|_| { + InputReaderError::Provider("InputAdded result offset exceeds u64".to_string()) + })?; + let expected = expected_start.checked_add(offset).ok_or_else(|| { + InputReaderError::Provider( + "InputAdded indices overflow the safe-input index space".to_string(), + ) + })?; if index != expected { return Err(InputReaderError::InconsistentL1Response(format!( "non-contiguous InputBox index: expected {expected}, got {index} — \ - provider returned an incomplete InputAdded set (F5)" + provider returned an incomplete InputAdded set" ))); } } @@ -668,7 +742,7 @@ fn check_input_index_contiguity( /// Verify the InputBox's own input count at the scanned safe block matches the /// number of inputs we now hold (`received_total` = previously stored + just /// received). A mismatch means the `get_logs` response was an incomplete prefix -/// — typically a truncated tail a contiguity check alone cannot see (F5, see +/// — typically a truncated tail a contiguity check alone cannot see (see /// `docs/threat-model/README.md`). Retryable [`InconsistentL1Response`]: a /// consistent provider returns the full set, and the matching count, on the /// next tick. @@ -682,14 +756,14 @@ fn check_input_count_complete( if onchain_count != received_total { return Err(InputReaderError::InconsistentL1Response(format!( "InputBox input count at block {safe_block} is {onchain_count}, expected \ - {received_total} — provider returned an incomplete get_logs set (F5)" + {received_total} — provider returned an incomplete get_logs set" ))); } Ok(()) } -/// The InputBox's per-app input count, pinned at `block`. Used as the F5 -/// completeness witness — see [`check_input_count_complete`]. Pinning to the +/// The InputBox's per-app input count, pinned at `block`. Used as the +/// input-completeness witness — see [`check_input_count_complete`]. Pinning to the /// scanned safe block (not `latest`) keeps it consistent with the `get_logs` /// range and forces the serving node to have that block's state. async fn input_count_at_block( @@ -758,6 +832,7 @@ mod tests { db_path, Address::ZERO, test_timing(), + ProcessLock::test(), ) } @@ -777,7 +852,7 @@ mod tests { #[tokio::test] async fn start_then_request_shutdown_joins_with_ok() { let db_file = NamedTempFile::new().expect("temp file"); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let reader = test_reader( db_file.path().to_string_lossy().into_owned(), "http://127.0.0.1:0".to_string(), @@ -801,7 +876,7 @@ mod tests { require_anvil(); let anvil = Anvil::default().block_time(1).timeout(30_000).spawn(); - let shutdown = ShutdownSignal::default(); + let shutdown = RuntimeScope::default(); let db_file = NamedTempFile::new().expect("temp file"); let reader = test_reader( db_file.path().to_string_lossy().into_owned(), @@ -881,6 +956,7 @@ mod tests { db_file.path().to_string_lossy().into_owned(), Address::ZERO, test_timing(), + ProcessLock::test(), ); let provider = alloy::providers::ProviderBuilder::new() .connect(anvil.endpoint_url().to_string().as_str()) @@ -963,6 +1039,7 @@ mod tests { }, Address::ZERO, test_timing(), + ProcessLock::test(), ) .await; @@ -1094,6 +1171,16 @@ mod tests { ); } + #[test] + fn input_index_contiguity_rejects_index_space_overflow() { + let err = check_input_index_contiguity(u64::MAX, &[u64::MAX, u64::MAX]) + .expect_err("the expected index must not saturate at u64::MAX"); + assert!( + matches!(&err, InputReaderError::Provider(m) if m.contains("overflow")), + "got {err:?}" + ); + } + #[test] fn input_count_complete_accepts_a_matching_count() { // Stored 7, received 3 more → 10 total; chain agrees → complete. @@ -1259,7 +1346,7 @@ mod tests { #[test] fn ingest_input_added_rejects_topic_payload_index_mismatch() { let tx = alloy_primitives::B256::repeat_byte(0xcc); - // The F5 witness reads the index from the topic; the watchdog reads it + // The input-completeness witness reads the index from the topic; the watchdog reads it // from the payload. The cross-check keeps the two provably aligned. let (event, log) = input_added_pair(5, evm_advance_payload(90, 6), Some(90), Some(tx)); let err = ingest_input_added(&event, &log).expect_err("index mismatch"); diff --git a/sequencer/src/l1/submitter/config.rs b/sequencer/src/l1/submitter/config.rs index beddd9dd..ab547a16 100644 --- a/sequencer/src/l1/submitter/config.rs +++ b/sequencer/src/l1/submitter/config.rs @@ -5,12 +5,14 @@ use std::time::Duration; /// Batch-submitter-specific options. L1 RPC URL and InputBox address are shared /// with the input reader and come from the same discovery at startup (see -/// `L1Config` in `config`). These fields are parsed as part of `RunConfig` and -/// passed through at runtime. +/// [`crate::l1::L1Config`] and its `identity`). These fields are parsed as +/// part of `RunConfig` and passed through at runtime. /// -/// Danger-zone tuning (`max_wait_blocks`, `preemptive_margin_blocks`, -/// `seconds_per_block`) lives in `ProtocolTiming`, not here — the submitter -/// doesn't read it. The [`crate::recovery::DangerDetector`] worker owns that. +/// Danger-zone tuning (`max_wait_blocks`, `preemptive_margin_blocks`) lives +/// in `ProtocolTiming`, not here — the submitter doesn't read it; the +/// [`crate::recovery::DangerDetector`] worker owns that. The one timing value +/// the poster does read is `seconds_per_block`, carried on +/// `BatchPosterConfig` for its confirmation timeout. #[derive(Debug, Clone)] pub struct BatchSubmitterConfig { /// How often the submitter polls for new work when idle. diff --git a/sequencer/src/l1/submitter/mod.rs b/sequencer/src/l1/submitter/mod.rs index 48178f8c..57ed3eb0 100644 --- a/sequencer/src/l1/submitter/mod.rs +++ b/sequencer/src/l1/submitter/mod.rs @@ -15,5 +15,7 @@ mod poster; mod worker; pub use config::BatchSubmitterConfig; -pub use poster::{BatchPoster, BatchPosterConfig, BatchPosterError, EthereumBatchPoster, TxHash}; -pub use worker::{BatchSubmitter, BatchSubmitterError, SubmitterExit}; +pub(crate) use poster::BatchPoster; +pub use poster::{BatchPosterConfig, BatchPosterError, EthereumBatchPoster, TxHash}; +pub(crate) use worker::BatchSubmitter; +pub use worker::{BatchSubmitterError, SubmitterExit}; diff --git a/sequencer/src/l1/submitter/poster.rs b/sequencer/src/l1/submitter/poster.rs index ff451b60..3801a5c7 100644 --- a/sequencer/src/l1/submitter/poster.rs +++ b/sequencer/src/l1/submitter/poster.rs @@ -9,12 +9,14 @@ use alloy::rpc::types::BlockNumberOrTag; use async_trait::async_trait; use cartesi_rollups_contracts::input_box::InputBox; use sequencer_core::batch::Batch; +use std::future::Future; use thiserror::Error; use tracing::{debug, info, warn}; use crate::l1::eip1559::{Eip1559Fees, estimate_fees}; use crate::l1::partition::{decode_evm_advance_input, get_input_added_events_ordered}; -use crate::l1::watermark::WalletNonceWatermarkSink; +use crate::l1::watermark::{WalletNonceWatermarkError, WalletNonceWatermarkSink}; +use crate::runtime::shutdown::RuntimeScope; pub type TxHash = alloy_primitives::B256; @@ -44,16 +46,48 @@ pub enum BatchPosterError { Provider(String), #[error("rpc chain id {rpc} does not match pinned chain id {expected}")] ChainIdMismatch { rpc: u64, expected: u64 }, + #[error( + "wallet nonce range starting at {first_nonce} for {batch_count} batches \ + cannot be represented durably" + )] + WalletNonceRangeUnrepresentable { + first_nonce: u64, + batch_count: usize, + }, + #[error("runtime stopped L1 submission after a persistent storage invariant failure")] + StorageInvariantViolation, + #[error("runtime shutdown cancelled L1 submission")] + Shutdown, + #[error(transparent)] + Watermark(#[from] WalletNonceWatermarkError), +} + +impl BatchPosterError { + pub(crate) fn is_terminal_invariant(&self) -> bool { + // Exhaustive on purpose: a new variant must decide its terminality + // here, not silently default to restartable. + match self { + Self::ChainIdMismatch { .. } + | Self::WalletNonceRangeUnrepresentable { .. } + | Self::StorageInvariantViolation => true, + Self::Watermark(source) => source.is_persistent_invariant(), + Self::Provider(_) | Self::Shutdown => false, + } + } } #[async_trait] -pub trait BatchPoster: Send + Sync { +pub(crate) trait BatchPoster: Send + Sync { /// Broadcast the payloads as L1 txs at consecutive wallet nonces. /// Implementations must raise `watermark` to the highest nonce they - /// are about to use *before* the first send (write-before-broadcast, - /// review R1a). + /// are about to use *before* the first send (write-before-broadcast). + /// Requires the externalization token: the caller consulted containment + /// this tick. Implementations may re-check at finer grain (the Ethereum + /// poster gates each send); a mock ignoring `_auth` is correct — the + /// token is the caller's proof, not the implementation's. async fn submit_batches( &self, + auth: crate::runtime::shutdown::Authorized<'_>, payloads: Vec>, watermark: &dyn WalletNonceWatermarkSink, ) -> Result, BatchPosterError>; @@ -68,11 +102,19 @@ pub trait BatchPoster: Send + Sync { pub struct EthereumBatchPoster { provider: DynProvider, config: BatchPosterConfig, + /// Externalization gate for keyed L1 sends. A construction-time field, + /// not a trait parameter: the gate is this implementation's posture, and + /// mocks were ignoring the parameter anyway. + shutdown: RuntimeScope, } impl EthereumBatchPoster { - pub fn new(provider: DynProvider, config: BatchPosterConfig) -> Self { - Self { provider, config } + pub fn new(provider: DynProvider, config: BatchPosterConfig, shutdown: RuntimeScope) -> Self { + Self { + provider, + config, + shutdown, + } } /// Conservative upper-bound timeout for waiting on confirmations, derived @@ -171,10 +213,46 @@ fn derive_confirmation_timeout( std::time::Duration::from_secs(blocks_to_wait.saturating_mul(seconds_per_block)) } +fn checked_highest_wallet_nonce( + first_nonce: u64, + batch_count: usize, +) -> Result { + let invalid = || BatchPosterError::WalletNonceRangeUnrepresentable { + first_nonce, + batch_count, + }; + let count = u64::try_from(batch_count).map_err(|_| invalid())?; + let last_offset = count.checked_sub(1).ok_or_else(invalid)?; + let highest = first_nonce.checked_add(last_offset).ok_or_else(invalid)?; + i64::try_from(highest).map_err(|_| invalid())?; + Ok(highest) +} + +async fn externalize_provider_call( + shutdown: &RuntimeScope, + call: impl Future>, +) -> Result { + if shutdown.is_storage_invariant_contained() { + return Err(BatchPosterError::StorageInvariantViolation); + } + tokio::select! { + biased; + _ = shutdown.wait_for_shutdown() => { + if shutdown.is_storage_invariant_contained() { + Err(BatchPosterError::StorageInvariantViolation) + } else { + Err(BatchPosterError::Shutdown) + } + } + result = call => result, + } +} + #[async_trait] impl BatchPoster for EthereumBatchPoster { async fn submit_batches( &self, + _auth: crate::runtime::shutdown::Authorized<'_>, payloads: Vec>, watermark: &dyn WalletNonceWatermarkSink, ) -> Result, BatchPosterError> { @@ -206,29 +284,38 @@ impl BatchPoster for EthereumBatchPoster { let fees = estimate_fees(&self.provider) .await .map_err(BatchPosterError::Provider)?; - let mut next_nonce = self.latest_account_nonce().await?; + let first_nonce = self.latest_account_nonce().await?; - // Write-before-broadcast (R1a): durably cover every nonce this + // Write-before-broadcast: durably cover every nonce this // tick will use before the first send. One raise to the highest // covers the whole consecutive range. - let highest_nonce = next_nonce.saturating_add(payloads.len() as u64 - 1); - watermark - .raise_to(highest_nonce) - .map_err(BatchPosterError::Provider)?; + let highest_nonce = checked_highest_wallet_nonce(first_nonce, payloads.len())?; + if self.shutdown.is_storage_invariant_contained() { + return Err(BatchPosterError::StorageInvariantViolation); + } + watermark.raise_to(highest_nonce)?; let mut tx_hashes = Vec::with_capacity(payloads.len()); - for payload in payloads { - let pending = self.send_batch_at_nonce(payload, next_nonce, &fees).await?; + for (offset, payload) in payloads.into_iter().enumerate() { + let offset = + u64::try_from(offset).expect("validated wallet-nonce range offset must fit in u64"); + let nonce = first_nonce + .checked_add(offset) + .expect("validated wallet-nonce range must not overflow"); + let pending = externalize_provider_call( + &self.shutdown, + self.send_batch_at_nonce(payload, nonce, &fees), + ) + .await?; let tx_hash = *pending.tx_hash(); debug!( - tx_nonce = next_nonce, + tx_nonce = nonce, %tx_hash, confirmation_depth = self.config.confirmation_depth, "sent batch submission tx to L1" ); tx_hashes.push(tx_hash); - next_nonce = next_nonce.saturating_add(1); } self.wait_for_confirmations(tx_hashes.as_slice()).await?; @@ -332,6 +419,7 @@ pub(crate) mod mock { impl BatchPoster for MockBatchPoster { async fn submit_batches( &self, + _auth: crate::runtime::shutdown::Authorized<'_>, payloads: Vec>, _watermark: &dyn WalletNonceWatermarkSink, ) -> Result, BatchPosterError> { @@ -379,13 +467,86 @@ mod tests { use super::{ BatchPoster, BatchPosterConfig, BatchPosterError, EthereumBatchPoster, - derive_confirmation_timeout, mock::MockBatchPoster, + checked_highest_wallet_nonce, derive_confirmation_timeout, externalize_provider_call, + mock::MockBatchPoster, }; - use crate::l1::watermark::WalletNonceWatermarkSink; + use crate::l1::watermark::{WalletNonceWatermarkError, WalletNonceWatermarkSink}; + use crate::runtime::shutdown::RuntimeScope; use alloy::node_bindings::Anvil; use alloy::providers::Provider; use alloy::rpc::types::BlockNumberOrTag; + #[test] + fn wallet_nonce_range_is_checked_before_watermark_or_broadcast() { + assert_eq!( + checked_highest_wallet_nonce(i64::MAX as u64 - 1, 2).expect("representable range"), + i64::MAX as u64 + ); + assert!(matches!( + checked_highest_wallet_nonce(i64::MAX as u64, 2), + Err(BatchPosterError::WalletNonceRangeUnrepresentable { .. }) + )); + assert!(matches!( + checked_highest_wallet_nonce(u64::MAX, 2), + Err(BatchPosterError::WalletNonceRangeUnrepresentable { .. }) + )); + } + + #[tokio::test] + async fn ordinary_shutdown_cancels_provider_send_without_terminal_classification() { + let shutdown = RuntimeScope::default(); + let (started_tx, started_rx) = tokio::sync::oneshot::channel(); + let send = tokio::spawn({ + let shutdown = shutdown.clone(); + async move { + externalize_provider_call(&shutdown, async move { + started_tx.send(()).expect("mark provider send started"); + std::future::pending::>().await + }) + .await + } + }); + started_rx.await.expect("provider send acquired the gate"); + + shutdown.request_shutdown(); + + assert!(matches!( + send.await.expect("provider send task"), + Err(BatchPosterError::Shutdown) + )); + assert!( + !shutdown.is_storage_invariant_contained(), + "ordinary shutdown cancellation must remain nonterminal" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn terminal_close_cancels_provider_send_and_finishes_publication() { + let shutdown = RuntimeScope::default(); + let (started_tx, started_rx) = tokio::sync::oneshot::channel(); + let send = tokio::spawn({ + let shutdown = shutdown.clone(); + async move { + externalize_provider_call(&shutdown, async move { + started_tx.send(()).expect("mark provider send started"); + std::future::pending::>().await + }) + .await + } + }); + started_rx.await.expect("provider send acquired the gate"); + + // Containment is sync and never waits — a stalled provider cannot + // delay the durable verdict. + shutdown.contain_storage_invariant_failure("test fault"); + + assert!(matches!( + send.await.expect("provider send task"), + Err(BatchPosterError::StorageInvariantViolation) + )); + assert!(shutdown.is_storage_invariant_contained()); + } + fn require_anvil() { assert!( std::process::Command::new("anvil") @@ -427,22 +588,24 @@ mod tests { } impl WalletNonceWatermarkSink for RecordingWatermarkSink { - fn raise_to(&self, highest: u64) -> Result<(), String> { + fn raise_to(&self, highest: u64) -> Result<(), WalletNonceWatermarkError> { self.calls.lock().expect("lock").push(highest); if self.fail { - Err("recording sink: forced failure".to_string()) + Err(WalletNonceWatermarkError::Other( + "recording sink: forced failure".to_string(), + )) } else { Ok(()) } } } - /// R1a write-before-broadcast: `submit_batches` must raise the watermark to + /// Write-before-broadcast: `submit_batches` must raise the watermark to /// cover the whole consecutive nonce range *before* the first send. We lock /// it with a sink that fails on `raise_to`: a correct poster aborts the tick /// before broadcasting anything, so the submitter's pending nonce is /// unchanged. If `raise_to` were moved after the first `addInput` send - /// (re-opening the F1 zombie-tx hole), that send would bump the pending + /// (re-opening the zombie-tx hole), that send would bump the pending /// nonce and this test would go red. Also pins the raise count (once) and /// value (`base + payloads.len() - 1`). (Mutation-checked: moving the raise /// after the send loop fails this test.) @@ -466,7 +629,7 @@ mod tests { long_block_range_error_codes: vec![], expected_chain_id: anvil.chain_id(), }; - let poster = EthereumBatchPoster::new(provider.clone(), config); + let poster = EthereumBatchPoster::new(provider.clone(), config, RuntimeScope::default()); let base_nonce = provider .get_transaction_count(submitter) @@ -475,10 +638,18 @@ mod tests { let sink = RecordingWatermarkSink::failing(); let payloads = vec![vec![0u8; 4], vec![1u8; 4], vec![2u8; 4]]; // 3 consecutive nonces - let result = poster.submit_batches(payloads, &sink).await; + let scope = RuntimeScope::default(); + let result = poster + .submit_batches(scope.authorize().expect("clear scope"), payloads, &sink) + .await; assert!( - matches!(result, Err(BatchPosterError::Provider(_))), + matches!( + result, + Err(BatchPosterError::Watermark( + WalletNonceWatermarkError::Other(_) + )) + ), "a failing watermark sink must abort submit_batches, got {result:?}" ); // (a) raised exactly once, (b) to the highest nonce of the range. @@ -524,7 +695,7 @@ mod tests { long_block_range_error_codes: vec![], expected_chain_id: wrong_chain_id, }; - let poster = EthereumBatchPoster::new(provider.clone(), config); + let poster = EthereumBatchPoster::new(provider.clone(), config, RuntimeScope::default()); let base_nonce = provider .get_transaction_count(submitter) @@ -536,7 +707,10 @@ mod tests { let sink = RecordingWatermarkSink::passing(); let payloads = vec![vec![0u8; 4], vec![1u8; 4]]; - let result = poster.submit_batches(payloads, &sink).await; + let scope = RuntimeScope::default(); + let result = poster + .submit_batches(scope.authorize().expect("clear scope"), payloads, &sink) + .await; assert!( matches!( @@ -561,6 +735,63 @@ mod tests { ); } + #[tokio::test] + async fn terminal_storage_fault_blocks_watermark_and_broadcast() { + require_anvil(); + let anvil = Anvil::default().spawn(); + let key = "0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80"; + let submitter = alloy_primitives::address!("0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266"); + let provider = crate::l1::provider::create_signer_provider(&anvil.endpoint(), key, false) + .expect("signer provider"); + let shutdown = RuntimeScope::default(); + let poster = EthereumBatchPoster::new( + provider.clone(), + BatchPosterConfig { + l1_submit_address: alloy_primitives::Address::repeat_byte(0x11), + app_address: alloy_primitives::Address::repeat_byte(0x22), + batch_submitter_address: submitter, + start_block: 0, + confirmation_depth: 0, + seconds_per_block: 1, + long_block_range_error_codes: vec![], + expected_chain_id: anvil.chain_id(), + }, + shutdown.clone(), + ); + let base_nonce = provider + .get_transaction_count(submitter) + .await + .expect("base nonce"); + let sink = RecordingWatermarkSink::passing(); + // Mint the token BEFORE the fault is contained: the honest race the + // ADR accepts. The poster's inner per-send gate must still refuse a + // stale token's send. + let auth = shutdown + .authorize() + .expect("token minted before the fault is contained"); + shutdown.contain_storage_invariant_failure("test fault"); + + let result = poster.submit_batches(auth, vec![vec![0u8; 4]], &sink).await; + + assert!(matches!( + result, + Err(BatchPosterError::StorageInvariantViolation) + )); + assert!( + sink.calls().is_empty(), + "terminal gate must close before the watermark write" + ); + let pending = provider + .get_transaction_count(submitter) + .block_id(BlockNumberOrTag::Pending.into()) + .await + .expect("pending nonce"); + assert_eq!( + pending, base_nonce, + "terminal gate must prevent an L1 broadcast" + ); + } + #[tokio::test] async fn mock_poster_tracks_requested_suffix_start_block() { let poster = MockBatchPoster::new(); diff --git a/sequencer/src/l1/submitter/worker.rs b/sequencer/src/l1/submitter/worker.rs index 5ea17639..168011ad 100644 --- a/sequencer/src/l1/submitter/worker.rs +++ b/sequencer/src/l1/submitter/worker.rs @@ -31,7 +31,8 @@ use thiserror::Error; use tracing::{debug, error}; use crate::l1::submitter::{BatchPoster, BatchPosterError, BatchSubmitterConfig}; -use crate::runtime::shutdown::ShutdownSignal; +use crate::runtime::process_lock::{ProcessLock, spawn_blocking_with_lock}; +use crate::runtime::shutdown::RuntimeScope; use crate::storage::{PendingBatch, Storage, StorageOpenError, SubmitterFrontier}; #[derive(Debug, Error)] @@ -40,12 +41,29 @@ pub enum BatchSubmitterError { OpenStorage(#[from] StorageOpenError), #[error(transparent)] Storage(#[from] rusqlite::Error), + #[error("storage task panicked while {operation}: persistent invariant failure")] + StorageTaskPanicked { operation: &'static str }, #[error("batch submitter join error: {0}")] Join(String), #[error(transparent)] Poster(#[from] BatchPosterError), } +impl BatchSubmitterError { + /// Whether this error poisons the run rather than restarting. Named + /// arms, no wildcard: a new variant must classify itself here. + pub(crate) fn is_terminal_invariant(&self) -> bool { + match self { + Self::Storage(source) => crate::storage::is_persistent_storage_error(source), + Self::OpenStorage(source) => crate::storage::is_persistent_storage_open_error(source), + Self::StorageTaskPanicked { .. } => true, + Self::Poster(source) => source.is_terminal_invariant(), + // A non-panic inner-task join is shutdown-path cancellation. + Self::Join(_) => false, + } + } +} + /// How the submitter loop exited. /// /// There is only one deliberate exit path (shutdown). Danger detection lives @@ -84,71 +102,93 @@ fn decide_submit_start(frontier: SubmitterFrontier, recently_observed_nonces: &[ ) } -pub struct BatchSubmitter { +pub(crate) struct BatchSubmitter { db_path: String, poster: Arc

, idle_poll_interval: Duration, - /// Write-before-broadcast hook (review R1a): the poster raises the + /// Write-before-broadcast hook: the poster raises the /// persisted wallet-nonce watermark through this before every send. watermark_sink: crate::l1::watermark::StorageWatermarkSink, + /// Retains data-directory exclusivity in detached blocking reads. + /// Required at construction: a submitter without data-dir ownership is + /// unrepresentable. + process_lock: ProcessLock, } impl BatchSubmitter

{ - pub fn new(db_path: impl Into, poster: Arc

, config: BatchSubmitterConfig) -> Self { + pub(crate) fn new( + db_path: impl Into, + poster: Arc

, + config: BatchSubmitterConfig, + process_lock: ProcessLock, + ) -> Self { let db_path = db_path.into(); Self { watermark_sink: crate::l1::watermark::StorageWatermarkSink::new(db_path.clone()), db_path, poster, idle_poll_interval: config.idle_poll_interval(), + process_lock, } } /// Spawn the worker loop. The `shutdown` signal is what the loop respects; /// passing it at start time (instead of construction time) keeps the /// construction phase pure. - pub fn start( + #[cfg(test)] + pub(crate) fn start( self, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, ) -> Result>, StorageOpenError> { + self.preflight_storage()?; + Ok(self.start_preflighted(shutdown)) + } + + /// Validate the storage dependency without starting a task. + pub(crate) fn preflight_storage(&self) -> Result<(), StorageOpenError> { let _ = Storage::open_read_only(self.db_path.as_str())?; - Ok(tokio::spawn( - async move { self.run_forever(shutdown).await }, - )) + Ok(()) } - /// Top-level driver. Races the work loop against the shutdown signal. - /// - /// `biased;` polls the shutdown arm first on every wakeup so a concurrent - /// shutdown wins over an in-flight `run_loop` step. Without `biased`, - /// `select!` would pick randomly between two ready branches and could - /// process one more iteration before shutting down. + /// Spawn after [`Self::preflight_storage`] succeeded. Infallible so the + /// runtime can launch all workers in one non-yielding ownership step. + pub(crate) fn start_preflighted( + self, + shutdown: RuntimeScope, + ) -> tokio::task::JoinHandle> { + tokio::spawn(async move { self.run_forever(shutdown).await }) + } + + /// Top-level driver. The biased shutdown arm promptly cancels async RPC + /// work; nested blocking DB jobs retain their own process-lock clone. async fn run_forever( self, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, ) -> Result { tokio::select! { biased; _ = shutdown.wait_for_shutdown() => Ok(SubmitterExit::Shutdown), - result = self.run_loop() => result, + result = self.run_loop(&shutdown) => result, } } /// Tick → sleep-if-idle → tick. Productive ticks re-enter immediately; /// idle or transient-error ticks wait `idle_poll_interval`. Fatal errors /// propagate. - async fn run_loop(&self) -> Result { + async fn run_loop(&self, scope: &RuntimeScope) -> Result { loop { - let outcome = match self.tick_once().await { + let outcome = match self.tick_once(scope).await { Ok(o) => o, - // A wrong-chain RPC is terminal — never retry-loop signing onto - // it. Lift it out of the transient `Poster` bucket below. - Err(e @ BatchSubmitterError::Poster(BatchPosterError::ChainIdMismatch { .. })) => { - error!(error = %e, "RPC serves the wrong chain — refusing to submit"); - return Err(e); - } Err(BatchSubmitterError::Poster(source)) => { + if source.is_terminal_invariant() { + let error = BatchSubmitterError::Poster(source); + error!( + error = %error, + "terminal batch-submitter input — refusing to submit" + ); + return Err(error); + } error!(error = %source, "L1 provider error — will retry"); TickOutcome::Transient } @@ -163,7 +203,10 @@ impl BatchSubmitter

{ } } - pub(crate) async fn tick_once(&self) -> Result { + pub(crate) async fn tick_once( + &self, + scope: &RuntimeScope, + ) -> Result { let frontier = self.load_frontier().await?; // Must start scanning at `safe_block + 1`: after a danger-zone shutdown @@ -171,9 +214,13 @@ impl BatchSubmitter

{ // slots backed by blocks at or below the safe head are already // resolved and folded into `accepted_next_nonce`. Re-scanning those // blocks here would double-count the finalized prefix. + let scan_start = frontier + .safe_block + .checked_add(1) + .expect("persisted safe block must leave room for the next block"); let recent_observed = self .poster - .observed_submitted_batch_nonces(frontier.safe_block.saturating_add(1)) + .observed_submitted_batch_nonces(scan_start) .await?; let from_nonce = decide_submit_start(frontier, &recent_observed); @@ -191,9 +238,16 @@ impl BatchSubmitter

{ } let submitted_count = pending.len(); let payloads: Vec> = pending.into_iter().map(|b| b.encoded).collect(); + // The L1 send requires the externalization token; the poster's own + // per-send gate stays as the bounded-lag re-check inside. + let Some(auth) = scope.authorize() else { + return Err(BatchSubmitterError::Poster( + BatchPosterError::StorageInvariantViolation, + )); + }; let tx_hashes = self .poster - .submit_batches(payloads, &self.watermark_sink) + .submit_batches(auth, payloads, &self.watermark_sink) .await?; if tx_hashes.len() != submitted_count { return Err(BatchSubmitterError::Poster(BatchPosterError::Provider( @@ -209,14 +263,15 @@ impl BatchSubmitter

{ async fn load_frontier(&self) -> Result { let db_path = self.db_path.clone(); - tokio::task::spawn_blocking(move || { + let process_lock = self.process_lock.clone(); + spawn_blocking_with_lock(process_lock, move || { let mut storage = Storage::open_read_only(&db_path)?; storage .submitter_frontier() .map_err(BatchSubmitterError::from) }) .await - .map_err(|err| BatchSubmitterError::Join(err.to_string()))? + .map_err(|err| map_storage_task_join(err, "loading the submitter frontier"))? } async fn pending_batches( @@ -224,25 +279,46 @@ impl BatchSubmitter

{ min_nonce: u64, ) -> Result, BatchSubmitterError> { let db_path = self.db_path.clone(); - tokio::task::spawn_blocking(move || { + let process_lock = self.process_lock.clone(); + spawn_blocking_with_lock(process_lock, move || { let mut storage = Storage::open_read_only(&db_path)?; storage .pending_batches(min_nonce) .map_err(BatchSubmitterError::from) }) .await - .map_err(|err| BatchSubmitterError::Join(err.to_string()))? + .map_err(|err| map_storage_task_join(err, "loading pending batches"))? + } +} + +/// Deliberately per-worker, not shared with the snapshot endpoint's +/// `storage_task`: this worker carries a typed error to the supervisor +/// through its exit channel, while an HTTP handler must contain immediately. +fn map_storage_task_join( + err: tokio::task::JoinError, + operation: &'static str, +) -> BatchSubmitterError { + if err.is_panic() { + BatchSubmitterError::StorageTaskPanicked { operation } + } else { + BatchSubmitterError::Join(err.to_string()) } } #[cfg(test)] mod tests { - use std::sync::Arc; + use std::sync::{Arc, Mutex}; - use alloy_primitives::Address; + use alloy_primitives::{Address, TxHash}; + use async_trait::async_trait; use super::{TickOutcome, decide_submit_start}; - use crate::l1::submitter::{BatchSubmitterConfig, poster::mock::MockBatchPoster}; + use crate::l1::submitter::{ + BatchPoster, BatchPosterError, BatchSubmitterConfig, poster::mock::MockBatchPoster, + }; + use crate::l1::watermark::WalletNonceWatermarkSink; + use crate::runtime::process_lock::ProcessLock; + use crate::runtime::shutdown::RuntimeScope; use crate::storage::test_helpers::{TestDb, temp_db}; use crate::storage::{SafeInputRange, Storage, StoredSafeInput, SubmitterFrontier}; use sequencer_core::protocol::ProtocolTiming; @@ -289,7 +365,7 @@ mod tests { fn seed_safe_submitted_batches(db_path: &str, safe_block: u64, nonces: &[u64]) { let mut storage = Storage::open(db_path).expect("open storage"); // Landings carry the local batch's real wire bytes so the - // content-identity check (review R2) accepts them. + // content-identity check accepts them. let inputs: Vec<_> = nonces .iter() .map(|nonce| StoredSafeInput { @@ -308,16 +384,72 @@ mod tests { .expect("append safe submitted batches"); } + struct BlockingObservedPoster { + started: Mutex>>, + } + + #[async_trait] + impl BatchPoster for BlockingObservedPoster { + async fn submit_batches( + &self, + _auth: crate::runtime::shutdown::Authorized<'_>, + _payloads: Vec>, + _watermark: &dyn WalletNonceWatermarkSink, + ) -> Result, BatchPosterError> { + unreachable!("the observed-nonce call never completes") + } + + async fn observed_submitted_batch_nonces( + &self, + _from_block: u64, + ) -> Result, BatchPosterError> { + if let Some(started) = self.started.lock().expect("lock").take() { + let _ = started.send(()); + } + std::future::pending().await + } + } + + #[tokio::test] + async fn shutdown_promptly_cancels_a_blocked_submitter_tick() { + let TestDb { _dir, path } = temp_db("submitter-cancel-mid-tick"); + seed_two_closed_batches(&path); + let (started_tx, started_rx) = tokio::sync::oneshot::channel(); + let poster = Arc::new(BlockingObservedPoster { + started: Mutex::new(Some(started_tx)), + }); + let submitter = + super::BatchSubmitter::new(path, poster, default_test_config(), ProcessLock::test()); + let shutdown = RuntimeScope::default(); + let task = submitter.start(shutdown.clone()).expect("start submitter"); + started_rx.await.expect("tick reached blocked RPC"); + + shutdown.request_shutdown(); + let result = tokio::time::timeout(std::time::Duration::from_millis(100), task) + .await + .expect("shutdown must not wait for the blocked RPC") + .expect("submitter task must join") + .expect("ordinary shutdown is clean"); + assert!(matches!(result, super::SubmitterExit::Shutdown)); + } + #[tokio::test] async fn tick_once_submits_first_missing_closed_batch() { let TestDb { _dir, path } = temp_db("tick-submits"); seed_two_closed_batches(&path); let mock = Arc::new(MockBatchPoster::new()); - let submitter = - super::BatchSubmitter::new(path.clone(), mock.clone(), default_test_config()); + let submitter = super::BatchSubmitter::new( + path.clone(), + mock.clone(), + default_test_config(), + ProcessLock::test(), + ); - let outcome = submitter.tick_once().await.expect("tick once"); + let outcome = submitter + .tick_once(&RuntimeScope::default()) + .await + .expect("tick once"); assert_eq!(outcome, TickOutcome::Submitted(3)); let submissions = mock.submissions(); @@ -335,10 +467,17 @@ mod tests { let mock = Arc::new(MockBatchPoster::new()); mock.set_observed_submitted_nonces(vec![2]); - let submitter = - super::BatchSubmitter::new(path.clone(), mock.clone(), default_test_config()); + let submitter = super::BatchSubmitter::new( + path.clone(), + mock.clone(), + default_test_config(), + ProcessLock::test(), + ); - let outcome = submitter.tick_once().await.expect("tick once"); + let outcome = submitter + .tick_once(&RuntimeScope::default()) + .await + .expect("tick once"); assert_eq!(outcome, TickOutcome::Idle); assert!(mock.submissions().is_empty()); assert_eq!(mock.last_from_block(), Some(11)); @@ -351,10 +490,17 @@ mod tests { seed_safe_submitted_batches(&path, 10, &[0, 1, 2]); let mock = Arc::new(MockBatchPoster::new()); - let submitter = - super::BatchSubmitter::new(path.clone(), mock.clone(), default_test_config()); + let submitter = super::BatchSubmitter::new( + path.clone(), + mock.clone(), + default_test_config(), + ProcessLock::test(), + ); - let outcome = submitter.tick_once().await.expect("tick once"); + let outcome = submitter + .tick_once(&RuntimeScope::default()) + .await + .expect("tick once"); assert_eq!(outcome, TickOutcome::Idle); assert!(mock.submissions().is_empty()); } @@ -366,10 +512,17 @@ mod tests { seed_safe_submitted_batches(&path, 10, &[0, 1]); let mock = Arc::new(MockBatchPoster::new()); - let submitter = - super::BatchSubmitter::new(path.clone(), mock.clone(), default_test_config()); + let submitter = super::BatchSubmitter::new( + path.clone(), + mock.clone(), + default_test_config(), + ProcessLock::test(), + ); - let outcome = submitter.tick_once().await.expect("tick once"); + let outcome = submitter + .tick_once(&RuntimeScope::default()) + .await + .expect("tick once"); assert_eq!(outcome, TickOutcome::Submitted(1)); assert_eq!(mock.last_from_block(), Some(11)); @@ -386,10 +539,17 @@ mod tests { let mock = Arc::new(MockBatchPoster::new()); mock.set_observed_submitted_nonces(vec![1]); - let submitter = - super::BatchSubmitter::new(path.clone(), mock.clone(), default_test_config()); + let submitter = super::BatchSubmitter::new( + path.clone(), + mock.clone(), + default_test_config(), + ProcessLock::test(), + ); - let outcome = submitter.tick_once().await.expect("tick once"); + let outcome = submitter + .tick_once(&RuntimeScope::default()) + .await + .expect("tick once"); assert_eq!(outcome, TickOutcome::Submitted(1)); assert_eq!(mock.last_from_block(), Some(11)); @@ -405,10 +565,11 @@ mod tests { let mock = Arc::new(MockBatchPoster::new()); mock.set_observed_submitted_error(Some("rpc fail")); - let submitter = super::BatchSubmitter::new(path, mock, default_test_config()); + let submitter = + super::BatchSubmitter::new(path, mock, default_test_config(), ProcessLock::test()); let err = submitter - .tick_once() + .tick_once(&RuntimeScope::default()) .await .expect_err("poster error should propagate"); assert!(matches!(err, super::BatchSubmitterError::Poster(_))); diff --git a/sequencer/src/l1/watermark.rs b/sequencer/src/l1/watermark.rs index 18fef189..ee937714 100644 --- a/sequencer/src/l1/watermark.rs +++ b/sequencer/src/l1/watermark.rs @@ -1,7 +1,7 @@ // (c) Cartesi and individual authors (see AUTHORS) // SPDX-License-Identifier: Apache-2.0 (see LICENSE) -//! Write-before-broadcast hook for the wallet-nonce watermark (review R1a). +//! Write-before-broadcast hook for the wallet-nonce watermark. //! //! Every component that broadcasts a transaction from the batch-submitter key //! — the batch poster and the mempool flusher's no-ops alike — must first @@ -9,19 +9,43 @@ //! and only then send. One uniform rule, no case analysis: the invariant is //! simply "the watermark covers the nonce of everything we ever sent", which //! is what lets the flush consume every slot we ever used without trusting -//! the local node's volatile mempool memory (the F1 zombie counterexample). +//! the local node's volatile mempool memory (the zombie counterexample). //! //! A crash between the commit and the send only over-covers: the flush later //! no-ops a never-used slot — one wasted no-op, harmless. -use crate::storage::Storage; +use thiserror::Error; + +use crate::storage::{ + Storage, StorageOpenError, is_persistent_storage_error, is_persistent_storage_open_error, +}; + +#[derive(Debug, Error)] +pub enum WalletNonceWatermarkError { + #[error("watermark sink could not open storage")] + OpenStorage(#[source] StorageOpenError), + #[error("watermark sink storage write failed")] + Storage(#[source] rusqlite::Error), + #[error("watermark sink failed: {0}")] + Other(String), +} + +impl WalletNonceWatermarkError { + pub(crate) fn is_persistent_invariant(&self) -> bool { + match self { + Self::OpenStorage(source) => is_persistent_storage_open_error(source), + Self::Storage(source) => is_persistent_storage_error(source), + Self::Other(_) => false, + } + } +} /// Durable raise of the wallet-nonce watermark, called *before* broadcasting. /// `raise_to(h)` must commit `watermark = max(watermark, h)` power-loss /// durably before returning; the caller may then broadcast txs at nonces /// `<= h`. pub trait WalletNonceWatermarkSink: Send + Sync { - fn raise_to(&self, highest: u64) -> Result<(), String>; + fn raise_to(&self, highest: u64) -> Result<(), WalletNonceWatermarkError>; } /// Sink backed by the sequencer DB's `wallet_nonce_watermark` singleton. @@ -40,11 +64,38 @@ impl StorageWatermarkSink { } impl WalletNonceWatermarkSink for StorageWatermarkSink { - fn raise_to(&self, highest: u64) -> Result<(), String> { - let mut storage = Storage::open_writer(&self.db_path) - .map_err(|e| format!("watermark sink: open storage: {e}"))?; + fn raise_to(&self, highest: u64) -> Result<(), WalletNonceWatermarkError> { + let mut storage = + Storage::open_writer(&self.db_path).map_err(WalletNonceWatermarkError::OpenStorage)?; storage .raise_wallet_nonce_watermark(highest) - .map_err(|e| format!("watermark sink: raise: {e}")) + .map_err(WalletNonceWatermarkError::Storage) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn persistent_classifier_preserves_storage_provenance() { + assert!( + WalletNonceWatermarkError::Storage(rusqlite::Error::QueryReturnedNoRows) + .is_persistent_invariant() + ); + assert!( + !WalletNonceWatermarkError::Storage(rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code: rusqlite::ffi::ErrorCode::DatabaseBusy, + extended_code: 5, + }, + None, + )) + .is_persistent_invariant() + ); + assert!( + !WalletNonceWatermarkError::Other("injected operational failure".into()) + .is_persistent_invariant() + ); } } diff --git a/sequencer/src/lib.rs b/sequencer/src/lib.rs index 120a99af..420746bc 100644 --- a/sequencer/src/lib.rs +++ b/sequencer/src/lib.rs @@ -10,12 +10,16 @@ //! - `l1` — input reader, batch submitter, L1 helpers //! - `storage` — SQLite-backed persistence (organized by writer role) //! - `recovery` — cascade invalidation + recovery batch -//! - `runtime` — orchestration, config, shutdown +//! - `commands` — the operator command brackets (run / setup / flush) plus +//! their command-scoped config and error taxonomy +//! - `runtime` — the runtime authority capabilities (process lock, scope) //! - `http` — shared HTTP error type + axum::serve orchestration //! //! The inclusion lane is the single writer of open-batch state; this is the //! invariant the storage layer relies on. +pub(crate) mod clock; +pub mod commands; pub mod egress; pub mod harness; pub mod http; @@ -25,7 +29,13 @@ pub mod recovery; pub mod runtime; pub mod storage; +#[cfg(test)] +extern crate self as sequencer; +#[cfg(test)] +mod integration_tests; + +pub use commands::config::{FlushConfig, RunConfig, SetupConfig}; +pub use commands::error::CommandError; +pub use commands::run::run; pub use harness::{Cli, Command, dispatch, run_main}; pub use http::{ApiConfig, ApiError, WS_CATCHUP_WINDOW_EXCEEDED_REASON}; -pub use runtime::config::{FlushConfig, RunConfig, SetupConfig}; -pub use runtime::{RunError, run}; diff --git a/sequencer/src/recovery/detector.rs b/sequencer/src/recovery/detector.rs index 1bd7c18d..361d06fb 100644 --- a/sequencer/src/recovery/detector.rs +++ b/sequencer/src/recovery/detector.rs @@ -6,8 +6,8 @@ //! A tiny background task that, every `poll_interval`, asks [`Storage::check_danger`] //! whether recovery or refusal is needed. If so, the task exits with //! [`DetectorExit::RecoveryRequired`] — the runtime turns that into a -//! deliberate non-error process shutdown, the orchestrator respawns, and -//! `run_preemptive_recovery` takes over on startup. +//! deliberate non-error process shutdown, the orchestrator respawns, and the +//! local-first startup reducer takes over. //! //! This is its own worker (not part of the batch submitter) because the two //! concerns are orthogonal: the submitter makes progress on L1, which involves @@ -24,22 +24,23 @@ use std::time::Duration; use thiserror::Error; use tracing::debug; -use crate::runtime::clock::unix_now_ms; -use crate::runtime::shutdown::ShutdownSignal; +use crate::clock::unix_now_ms; +use crate::runtime::process_lock::{ProcessLock, spawn_blocking_with_lock}; +use crate::runtime::shutdown::RuntimeScope; use crate::storage::{DangerStatus, Storage, StorageOpenError}; use sequencer_core::protocol::ProtocolTiming; /// How the detector's loop exited. /// /// `RecoveryRequired` is a *deliberate* exit — not an error. The runtime maps -/// it to a distinct `RunError` variant so operators can tell "time to recover +/// it to a distinct `CommandError` variant so operators can tell "time to recover /// or refuse startup" apart from "something crashed". #[derive(Debug)] pub enum DetectorExit { /// Shutdown signal fired before any danger was detected. Shutdown, - /// A non-safe danger status was observed. Stop and let startup dispatch - /// the recovery/refusal path from a fresh read. + /// A non-safe danger status was observed. Stop and let startup re-derive + /// the recovery/refusal path from fresh facts. RecoveryRequired { status: DangerStatus }, } @@ -49,52 +50,83 @@ pub enum DangerDetectorError { OpenStorage(#[from] StorageOpenError), #[error(transparent)] Storage(#[from] rusqlite::Error), + #[error("storage task panicked while checking danger: persistent invariant failure")] + StorageTaskPanicked, #[error("danger detector join error: {0}")] Join(String), } +impl DangerDetectorError { + /// Whether this error poisons the run rather than restarting. Named + /// arms, no wildcard: a new variant must classify itself here. + pub(crate) fn is_terminal_invariant(&self) -> bool { + match self { + Self::Storage(source) => crate::storage::is_persistent_storage_error(source), + Self::OpenStorage(source) => crate::storage::is_persistent_storage_open_error(source), + Self::StorageTaskPanicked => true, + // A non-panic inner-task join is shutdown-path cancellation. + Self::Join(_) => false, + } + } +} + pub struct DangerDetector { db_path: String, protocol: ProtocolTiming, poll_interval: Duration, + /// Retains data-directory exclusivity in detached blocking checks. + /// Required at construction. + process_lock: ProcessLock, } impl DangerDetector { - pub fn new( + pub(crate) fn new( db_path: impl Into, protocol: ProtocolTiming, poll_interval: Duration, + process_lock: ProcessLock, ) -> Self { Self { db_path: db_path.into(), protocol, poll_interval, + process_lock, } } /// Spawn the detector loop. The `shutdown` signal is what the loop /// respects; passing it at start time (instead of construction time) keeps /// the construction phase pure. - pub fn start( + #[cfg(test)] + pub(crate) fn start( self, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, ) -> Result>, StorageOpenError> { + self.preflight_storage()?; + Ok(self.start_preflighted(shutdown)) + } + + /// Validate the storage dependency without starting a task. + pub(crate) fn preflight_storage(&self) -> Result<(), StorageOpenError> { let _ = Storage::open_read_only(self.db_path.as_str())?; - Ok(tokio::spawn( - async move { self.run_forever(shutdown).await }, - )) + Ok(()) + } + + /// Spawn after [`Self::preflight_storage`] succeeded. Infallible so the + /// runtime can launch all workers in one non-yielding ownership step. + pub(crate) fn start_preflighted( + self, + shutdown: RuntimeScope, + ) -> tokio::task::JoinHandle> { + tokio::spawn(async move { self.run_forever(shutdown).await }) } - /// Top-level driver. Races the work loop against the shutdown signal. - /// - /// `biased;` polls the shutdown arm first on every wakeup so a concurrent - /// shutdown wins over an in-flight `run_loop` step. Without `biased`, - /// `select!` would pick randomly between two ready branches and could - /// process one more iteration before shutting down. + /// Top-level driver. The biased shutdown arm returns promptly; an in-flight + /// blocking check retains its own process-lock clone until it stops. async fn run_forever( self, - shutdown: ShutdownSignal, + shutdown: RuntimeScope, ) -> Result { tokio::select! { biased; @@ -104,8 +136,7 @@ impl DangerDetector { } /// Tick → sleep → tick. Returns `RecoveryRequired` when a non-Safe danger - /// status fires. Shutdown is handled by the outer `run_forever` select, - /// so this loop has no shutdown concerns. + /// status fires. async fn run_loop(self) -> Result { loop { match self.check_once().await? { @@ -113,11 +144,9 @@ impl DangerDetector { debug!("danger check: safe"); } status => { - // All non-Safe variants exit for recovery/refusal. The - // dispatch difference (flush vs no-flush vs refuse) - // only matters at the next startup — `decide_startup_action` - // re-runs `check_danger` and routes based on which variant - // fires this time. + // All non-Safe variants exit. The reducer re-inspects on + // the next startup and selects one phase, Retry, or Refuse + // from the then-current facts. tracing::error!( ?status, danger_threshold = self.protocol.danger_threshold(), @@ -135,14 +164,23 @@ impl DangerDetector { let db_path = self.db_path.clone(); let protocol = self.protocol; let now_ms = unix_now_ms(); - tokio::task::spawn_blocking(move || { + let process_lock = self.process_lock.clone(); + spawn_blocking_with_lock(process_lock, move || { let mut storage = Storage::open_read_only(&db_path)?; storage .check_danger(&protocol, now_ms) .map_err(DangerDetectorError::from) }) .await - .map_err(|err| DangerDetectorError::Join(err.to_string()))? + .map_err(map_storage_task_join)? + } +} + +fn map_storage_task_join(err: tokio::task::JoinError) -> DangerDetectorError { + if err.is_panic() { + DangerDetectorError::StorageTaskPanicked + } else { + DangerDetectorError::Join(err.to_string()) } } @@ -174,9 +212,13 @@ mod tests { .expect("record fresh safe-head observation"); drop(storage); - let shutdown = ShutdownSignal::default(); - let detector = - DangerDetector::new(db.path.clone(), test_protocol(), Duration::from_millis(50)); + let shutdown = RuntimeScope::default(); + let detector = DangerDetector::new( + db.path.clone(), + test_protocol(), + Duration::from_millis(50), + ProcessLock::test(), + ); let handle = detector.start(shutdown.clone()).expect("start detector"); tokio::time::sleep(Duration::from_millis(20)).await; @@ -221,8 +263,13 @@ mod tests { .expect("append"); drop(storage); - let shutdown = ShutdownSignal::default(); - let detector = DangerDetector::new(db.path.clone(), protocol, Duration::from_millis(50)); + let shutdown = RuntimeScope::default(); + let detector = DangerDetector::new( + db.path.clone(), + protocol, + Duration::from_millis(50), + ProcessLock::test(), + ); let handle = detector.start(shutdown).expect("start detector"); let exit = tokio::time::timeout(Duration::from_secs(2), handle) @@ -283,7 +330,7 @@ mod tests { // Rewind synced_at_ms by 25 blocks' worth of wall-clock time so the // wall-clock arm shaves 25 off the threshold (1125 → 1100). At 1100, // batch 1's age = 1100 trips `>=`. Estimated batch danger fires. - let now_ms = crate::runtime::clock::unix_now_ms(); + let now_ms = crate::clock::unix_now_ms(); drop(storage); let rewind_conn = Storage::open_connection(&db.path).expect("open raw connection to rewind synced_at_ms"); @@ -295,8 +342,13 @@ mod tests { .expect("rewind safe-progress timestamp"); drop(rewind_conn); - let shutdown = ShutdownSignal::default(); - let detector = DangerDetector::new(db.path.clone(), protocol, Duration::from_millis(50)); + let shutdown = RuntimeScope::default(); + let detector = DangerDetector::new( + db.path.clone(), + protocol, + Duration::from_millis(50), + ProcessLock::test(), + ); let handle = detector.start(shutdown).expect("start detector"); let exit = tokio::time::timeout(Duration::from_secs(2), handle) diff --git a/sequencer/src/recovery/flusher.rs b/sequencer/src/recovery/flusher.rs index e7de31f3..b51b140a 100644 --- a/sequencer/src/recovery/flusher.rs +++ b/sequencer/src/recovery/flusher.rs @@ -19,12 +19,27 @@ use std::time::Duration; use thiserror::Error; use tracing::{debug, error, info}; -use crate::l1::watermark::{StorageWatermarkSink, WalletNonceWatermarkSink}; +use crate::l1::watermark::{ + StorageWatermarkSink, WalletNonceWatermarkError, WalletNonceWatermarkSink, +}; #[derive(Debug, Error)] pub enum FlushError { #[error("provider/transport: {0}")] Provider(String), + #[error(transparent)] + Watermark(#[from] WalletNonceWatermarkError), +} + +impl FlushError { + pub(crate) fn is_terminal_invariant(&self) -> bool { + // Exhaustive on purpose: a new variant must decide its terminality + // here, not silently default to restartable. + match self { + Self::Watermark(source) => source.is_persistent_invariant(), + Self::Provider(_) => false, + } + } } pub struct MempoolFlusher { @@ -41,7 +56,7 @@ pub struct MempoolFlusher { /// `safe_poll_interval` is one block — matches the natural cadence for /// `get_transaction_count(Safe)` to advance. /// -/// H6 regression: both values must scale with `CARTESI_SEQUENCER_SECONDS_PER_BLOCK`; a fixed +/// Both values must scale with `CARTESI_SEQUENCER_SECONDS_PER_BLOCK`; a fixed /// 12s assumption would mis-pace on non-mainnet chains. fn derive_timeouts(seconds_per_block: u64) -> (Duration, Duration) { ( @@ -102,7 +117,7 @@ impl MempoolFlusher { /// differ only in where the signing key, submitter address, and watermark /// come from, and in the surrounding error type — so callers resolve those /// (provider creation keeps each site's own error mapping; the returned - /// [`FlushError`] maps into `RunError`/`RecoveryError` via the existing + /// [`FlushError`] maps into `CommandError`/`RecoveryError` via the existing /// `From` impls) and this owns the sink build + `flush_and_wait`. pub(crate) async fn flush_to_safe( provider: DynProvider, @@ -130,9 +145,9 @@ impl MempoolFlusher { /// nonce slots, then waiting until every slot we ever used is safe. /// /// `watermark` is the persisted wallet-nonce watermark — the highest - /// nonce this deployment ever broadcast (review R1a), or `None` if - /// nothing was ever broadcast (or no DB survives, the cockroach-recovery - /// best-effort case, R1b). The loop runs until + /// nonce this deployment ever broadcast, or `None` if nothing was ever + /// broadcast (or no DB survives, the cockroach-recovery best-effort + /// case). The loop runs until /// /// ```text /// pending <= safe && safe >= watermark + 1 @@ -141,9 +156,9 @@ impl MempoolFlusher { /// The first conjunct resolves every slot the local node remembers; the /// second is the durable anchor — it refuses to declare victory until /// slot `watermark` is consumed at safe depth, covering zombie txs the - /// local node has forgotten but the network may still hold (the F1 - /// counterexample). It doubles as the post-flush assert from R1a: the - /// function cannot return success without it. + /// local node has forgotten but the network may still hold. It doubles + /// as the post-flush assert: the function cannot return success without + /// it. /// /// At each iteration: /// 1. Submit 0-ETH self-transfers for nonces in @@ -158,9 +173,9 @@ impl MempoolFlusher { /// 4. If any watch times out, retry the outer loop (tx may have been dropped, /// or the original batch may be making progress instead). /// - /// Returns the L1 **safe block number** at which resolution was observed - /// (review F2): the caller must not cascade until its own re-synced view - /// reaches at least this block. + /// Returns the L1 **safe block number** at which resolution was observed: + /// the caller must not cascade until its own re-synced view reaches at + /// least this block. pub async fn flush_and_wait( &self, watermark: Option, @@ -172,7 +187,10 @@ impl MempoolFlusher { let pending_nonce = self.nonce_at(BlockNumberOrTag::Pending).await?; // The durable anchor: every slot we ever used must be consumed // at safe depth, regardless of what the local pool remembers. - let required_safe_nonce = watermark.map_or(0, |w| w.saturating_add(1)); + let required_safe_nonce = watermark.map_or(0, |w| { + w.checked_add(1) + .expect("persisted wallet nonce watermark must leave room for the next nonce") + }); if pending_nonce <= safe_nonce && safe_nonce >= required_safe_nonce { let safe_block = self.safe_block_number().await?; @@ -186,7 +204,12 @@ impl MempoolFlusher { } let flush_end = pending_nonce.max(required_safe_nonce); - let unresolved = flush_end.saturating_sub(safe_nonce); + // The completion predicate failed, so either pending > safe or + // required_safe > safe. Therefore their maximum is strictly above + // safe; clamping here would hide a broken loop invariant. + let unresolved = flush_end + .checked_sub(safe_nonce) + .expect("incomplete flush must have at least one unresolved nonce"); if attempt == 0 { info!( @@ -219,8 +242,10 @@ impl MempoolFlusher { // about to send (a no-op above the current watermark can only // happen if someone else used our key — over-covering then is // exactly right). - sink.raise_to(flush_end.saturating_sub(1)) - .map_err(FlushError::Provider)?; + let highest_nonce = flush_end + .checked_sub(1) + .expect("a non-empty flush range must have a highest nonce"); + sink.raise_to(highest_nonce)?; } let tx_hashes = self.submit_noops(latest_nonce, flush_end).await?; @@ -363,12 +388,12 @@ mod tests { /// Sink for tests that don't assert on watermark raises. struct NoopWatermarkSink; impl WalletNonceWatermarkSink for NoopWatermarkSink { - fn raise_to(&self, _highest: u64) -> Result<(), String> { + fn raise_to(&self, _highest: u64) -> Result<(), WalletNonceWatermarkError> { Ok(()) } } - // ── H5: replacement-fee bump keeps no-ops competitive ───────── + // ── Replacement-fee bump keeps no-ops competitive ───────── #[test] fn replacement_fee_bump_exceeds_ten_percent_for_max_fee() { @@ -441,7 +466,7 @@ mod tests { assert_eq!(new_prio, u128::MAX); } - // ── H6: timeouts derive from seconds_per_block ──────────────── + // ── Timeouts derive from seconds_per_block ──────────────── #[test] fn timeouts_derive_from_seconds_per_block() { @@ -623,9 +648,9 @@ mod tests { let provider = signer_provider(&anvil); let addr = anvil.addresses()[0]; - // Models the F1 zombie: the persisted watermark says slot 0 was + // Models the zombie: the persisted watermark says slot 0 was // broadcast, but the local pool has no memory of it - // (pending == safe == 0). The pre-R1a `pending <= safe` early + // (pending == safe == 0). The pre-anchor `pending <= safe` early // return would declare victory immediately and leave the slot to // a zombie; the anchored flush must consume slot 0 with a no-op // and wait for it to reach safe depth. @@ -651,7 +676,7 @@ mod tests { ); assert!( observed_safe_block > 0, - "flush must report the safe block it observed resolution at (F2)" + "flush must report the safe block it observed resolution at" ); } diff --git a/sequencer/src/recovery/mod.rs b/sequencer/src/recovery/mod.rs index 98b84f75..ad42df3e 100644 --- a/sequencer/src/recovery/mod.rs +++ b/sequencer/src/recovery/mod.rs @@ -1,493 +1,1022 @@ // (c) Cartesi and individual authors (see AUTHORS) // SPDX-License-Identifier: Apache-2.0 (see LICENSE) -//! Preemptive recovery: detect danger zone, then recover via Tip invalidation -//! or post-flush cascade. +//! Run-start recovery authority. //! -//! At runtime a dedicated [`DangerDetector`] worker polls `Storage::check_danger` each -//! tick. If the L1 view is stale, a closed batch or Tip crosses -//! `danger_threshold`, or the batch-relative wall-clock estimate fires during -//! an L1 outage, the detector exits with `DetectorExit::RecoveryRequired`, the -//! runtime maps that to `DangerDetectorExit::DangerDetected` under -//! `RunError::Worker`, and the process exits. The detector tripping is *only* -//! a trigger to enter startup recovery/refusal — it doesn't make the cascade -//! decision. External orchestration restarts the sequencer, and this startup -//! path runs. +//! A pure reducer selects exactly one phase from one transactionally +//! consistent local inspection. The driver executes at most that phase and +//! always returns to local inspection before another phase or runtime +//! admission. Canonical divergence is therefore an absorbing local fact, not +//! a special check each effect must remember. //! -//! Startup recovery branches on [`decide_startup_action`]: -//! -//! - `FlushAndCascade`: a closed batch past gold is dangerous. Flush the mempool, -//! re-sync the safe head, then call [`crate::storage::Storage::recover_post_flush`] which cascades -//! everything past the gold frontier (every non-gold batch is doomed: -//! Silver-stale, Silver-poisoned, or Pending no-op'd). If all closed are gold, -//! falls through to a Tip danger-zone check — see `docs/recovery/README.md` Step 5. -//! - `RecoverTip`: only the open Tip is dangerous. It has no L1 footprint, so call -//! [`crate::storage::Storage::recover_aging_tip`] directly without flushing. -//! - `Proceed`: no danger detected. No DB writes here; the genesis Tip (on a -//! fresh DB) is opened by the structural [`crate::storage::Storage::ensure_open_tip`] -//! step in `Workers::spawn`, after recovery and before the lane starts. -//! - `Refuse`: L1 view is stale or batch-relative estimated danger fired; bail -//! out and surface to the operator. -//! -//! ## Fault model -//! -//! Recovery is designed to handle **submission and outage failures**: the sequencer -//! crashes, the L1 provider becomes unreachable, transactions are dropped from the -//! mempool, or the process is offline for an extended period. It is **not** designed -//! to handle arbitrarily malformed self-submissions. The scheduler frontier -//! reconstruction (`populate_safe_accepted_batches`) trusts that on-chain batches -//! from the sequencer's own address are structurally valid. This is a deliberate -//! system assumption, not a gap — the sequencer controls its own submissions. -//! -//! See `docs/recovery/` for the full design, TLA+ specs, and design history. +//! The flush and post-flush-sync witnesses live only for this boot attempt. A +//! crash loses them and the next boot repeats the idempotent flush; no durable +//! recovery-phase state machine is introduced. See `docs/recovery/README.md` +//! and `docs/recovery/admission.tla`. mod detector; mod flusher; use thiserror::Error; +use crate::l1::L1Config; use crate::l1::reader::{InputReader, InputReaderError}; -use crate::runtime::config::L1Config; -use crate::storage::{self, DangerStatus, StorageOpenError}; +use crate::storage::{ + self, DangerStatus, RecoveryInspection, RecoveryMutationError, StorageOpenError, +}; pub use detector::{DangerDetector, DangerDetectorError, DetectorExit}; pub use flusher::{FlushError, MempoolFlusher}; use sequencer_core::protocol::ProtocolTiming; +/// A startup recovery failure is already classified when it leaves the +/// controller. Runtime lifecycle settlement projects only this outer class; +/// it never reinterprets raw provider/storage/phase errors. #[derive(Debug, Error)] pub enum RecoveryError { + #[error("startup recovery should retry: {0}")] + Retry(Box), + #[error("startup recovery refused: {0}")] + Refuse(Box), +} + +/// Diagnostic provenance retained underneath the controller's retry/refuse +/// verdict. +#[derive(Debug, Error)] +pub enum RecoveryFailure { #[error(transparent)] - OpenStorage(#[from] StorageOpenError), + PolicyRetry(#[from] RecoveryRetryReason), #[error(transparent)] - Storage(#[from] rusqlite::Error), + PolicyRefusal(#[from] RecoveryRefusalReason), + #[error("open storage: {0}")] + OpenStorage(#[source] StorageOpenError), + #[error("storage: {0}")] + Storage(#[source] rusqlite::Error), #[error("flush: {0}")] - Flush(#[from] flusher::FlushError), + Flush(#[source] FlushError), #[error("input reader: {0}")] - InputReader(#[from] InputReaderError), + InputReader(#[source] InputReaderError), #[error("provider: {0}")] Provider(String), #[error("recovery flush chain-id mismatch: rpc {rpc} != pinned {expected}")] ChainIdMismatch { rpc: u64, expected: u64 }, - #[error("startup refused: {0:?}")] - Refuse(RefuseReason), +} + +#[derive(Debug, Error, Clone, Copy, PartialEq, Eq)] +pub enum RecoveryRetryReason { + #[error("the persisted L1 view is stale")] + L1ViewStale, + #[error("batch {batch_index} is in danger only under wall-clock estimation")] + EstimatedBatchInDanger { batch_index: u64 }, + #[error("danger persists after repair: {status:?}")] + DangerPersists { status: DangerStatus }, #[error( - "post-flush re-sync reached safe block {resynced_safe_block}, behind the \ - flusher's observed resolution at {flush_observed_safe_block}; refusing to \ - cascade on a lagging L1 view (respawn retries with a fresher view)" + "post-flush re-sync reached safe block {resynced_safe_block}, behind the flush observation at {flush_observed_safe_block}" )] ResyncBehindFlushView { resynced_safe_block: u64, flush_observed_safe_block: u64, }, + #[error("local recovery facts changed before phase execution: {status:?}")] + StaleDecision { status: DangerStatus }, + #[error("runtime preparation outlived its clean admission decision ({decision})")] + AdmissionChanged { decision: &'static str }, +} + +#[derive(Debug, Error, Clone, Copy, PartialEq, Eq)] +pub enum RecoveryRefusalReason { + /// A fully accepted L1 landing failed content identity. Standard recovery + /// assumes the opposite and is forbidden. + #[error("canonical divergence at batch nonce {nonce}")] + CanonicalDivergence { nonce: u64 }, + #[error("the completed setup has no finalized snapshot")] + MissingFinalizedSnapshot, + #[error("post-sync recovery has no persisted safe head")] + MissingSafeHead, +} + +/// Single-use proof that the run's final admission decision — the same pure +/// reducer over one transactionally consistent fact set — selected `Admit` +/// after all fallible preparation completed. Its private field makes +/// construction exclusive to [`admit_runtime`]; runtime code may consume the +/// proof but cannot mint one. +#[must_use = "runtime admission must be consumed by PreparedRuntime::launch"] +#[derive(Debug)] +pub(crate) struct RuntimeAdmission { + _private: (), +} + +impl RecoveryError { + pub(crate) fn retry(failure: impl Into) -> Self { + Self::Retry(Box::new(failure.into())) + } + + pub(crate) fn refuse(failure: impl Into) -> Self { + Self::Refuse(Box::new(failure.into())) + } + + pub(crate) fn is_retryable(&self) -> bool { + matches!(self, Self::Retry(_)) + } } -/// F2 coherence guard: refuse if the post-flush re-sync's safe head lags the -/// block the flusher observed resolution at. Folding (`setup --recovery`) or -/// cascading (runtime danger) on a view that stops short of the flush's -/// resolution would miss inputs the flush already settled. Shared by both -/// recovery paths; the orchestrator respawn retries with a fresher L1 view (the -/// flush is idempotent). See [`RecoveryError::ResyncBehindFlushView`]. +/// The post-flush resync coherence check — the resynced safe block must +/// reach the flush observation before cascade — shared with +/// `setup --recovery`. Runtime recovery also enforces this inside the +/// guarded cascade transaction. pub(crate) fn assert_resync_caught_up( resynced_safe_block: u64, flush_observed_safe_block: u64, ) -> Result<(), RecoveryError> { if resynced_safe_block < flush_observed_safe_block { - return Err(RecoveryError::ResyncBehindFlushView { - resynced_safe_block, - flush_observed_safe_block, - }); + return Err(RecoveryError::retry( + RecoveryRetryReason::ResyncBehindFlushView { + resynced_safe_block, + flush_observed_safe_block, + }, + )); } Ok(()) } -/// Why startup cannot proceed safely. -/// -/// Each variant captures a DB/L1-view state that makes recovery or normal -/// startup unsafe. The operator sees the variant in logs and must intervene. +/// The phase-ordering state machine of one boot attempt. `Flushed` and +/// `PostFlushSynced` carry the flush observation as an ephemeral, +/// memory-only witness: `drive_recovery` is its only writer, phases are its +/// only source, and it never persists — a restarted attempt has no witness +/// and must flush again. Cascade is therefore reachable only through +/// Flush → Sync *in this process* (the ADR's recovery-reducer mechanism). +/// This one enum is both the +/// reducer's input and the driver's completion type; the previous +/// `RecoveryState`/witness-struct/`PhaseCompletion` triple encoded the same +/// five variants three times. #[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum RefuseReason { - /// A fully-accepted L1 landing failed the content-identity check - /// (review R2): canonical state diverged from the local batch tree. - /// Terminal — no restart self-heals it; the operator must run cockroach - /// recovery (wipe + rebuild from L1). Standard recovery is forbidden: - /// it reconciles the tree's *shape* assuming accepted nonce N is our - /// batch N, which is exactly what no longer holds. - CanonicalDivergence { nonce: u64 }, - /// The L1 safe block timestamp is too old or unknown, so the local L1 view - /// is not usable for recovery or continued soft confirmations. - L1ViewStale, - /// Batch-relative wall-clock estimation says this batch consumed its - /// remaining runway, but the observed safe block has not crossed danger. - /// Refuse rather than recover from estimated state. - EstimatedBatchInDanger { batch_index: u64 }, +enum RecoveryProgress { + NeedInitialSync, + Inspecting, + Flushed { observed_safe_block: u64 }, + PostFlushSynced { required_safe_block: u64 }, + Repaired, } -/// What a fresh startup must do, given the current danger state. -/// -/// Pure function output — no side effects. The `run_preemptive_recovery` -/// driver executes the chosen action. -/// -/// The four non-Refuse variants encode the recovery split: -/// -/// - `Proceed`: no danger detected. No recovery work needed; the genesis Tip -/// (fresh DB) is opened by the structural `ensure_open_tip` step, not here. -/// - `RecoverTip`: aging Tip, no closed batch in danger. The Tip has no L1 -/// footprint, so we cascade it directly with no flush. -/// - `FlushAndCascade`: closed batch in danger. We need a flush to resolve -/// its L1 transaction's fate before the cascade decision. -/// - `Refuse`: can't proceed safely; surface to the operator. #[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum StartupAction { - /// No danger; no DB writes (genesis Tip handled by `ensure_open_tip`). - Proceed, - /// Open Tip is past `danger_threshold` and no closed batch is in danger. - /// No flush needed (Tip has no L1 slot to resolve); cascade the Tip - /// directly. - RecoverTip { batch_index: u64 }, - /// Closed batch past the gold frontier is in danger. Flush the mempool, - /// re-sync, then run the post-flush cascade. - FlushAndCascade { batch_index: u64 }, - /// Can't proceed safely; return the reason and let the operator decide. - Refuse(RefuseReason), +enum RecoveryPhase { + InitialSync, + EnsureOpenTip, + RecoverTip { expected_batch_index: u64 }, + Flush, + PostFlushSync { required_safe_block: u64 }, + Cascade { required_safe_block: u64 }, } -impl StartupAction { +impl RecoveryPhase { fn label(self) -> &'static str { match self { - StartupAction::Proceed => "proceed", - StartupAction::RecoverTip { .. } => "recover_tip", - StartupAction::FlushAndCascade { .. } => "flush_and_cascade", - StartupAction::Refuse(_) => "refuse", + Self::InitialSync => "initial_sync", + Self::EnsureOpenTip => "ensure_open_tip", + Self::RecoverTip { .. } => "recover_tip", + Self::Flush => "flush", + Self::PostFlushSync { .. } => "post_flush_sync", + Self::Cascade { .. } => "cascade", } } } -impl RefuseReason { - /// Stable label for logs/metrics. Inherent method, co-located with the - /// variants (`DangerStatus::label`/`batch_index` live next to that enum). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum RecoveryDecision { + Admit, + Act(RecoveryPhase), + Retry(RecoveryRetryReason), + Refuse(RecoveryRefusalReason), +} + +impl RecoveryDecision { fn label(self) -> &'static str { match self { - RefuseReason::CanonicalDivergence { .. } => "canonical_divergence", - RefuseReason::L1ViewStale => "l1_view_stale", - RefuseReason::EstimatedBatchInDanger { .. } => "estimated_batch_in_danger", + Self::Admit => "admit", + Self::Act(phase) => phase.label(), + Self::Retry(_) => "retry", + Self::Refuse(_) => "refuse", } } } -/// Pure decision: given the danger status, return what startup should do. L1 -/// reachability is an execution concern: if `FlushAndCascade` cannot reach L1, -/// the flusher returns an error and the orchestrator retries. -pub fn decide_startup_action(danger: DangerStatus) -> StartupAction { - match danger { - DangerStatus::Safe => StartupAction::Proceed, - DangerStatus::CanonicalDivergence(nonce) => { - StartupAction::Refuse(RefuseReason::CanonicalDivergence { nonce }) - } - DangerStatus::ClosedBatchInDanger(batch_index) => { - StartupAction::FlushAndCascade { batch_index } - } - DangerStatus::TipInDanger(batch_index) => StartupAction::RecoverTip { batch_index }, - DangerStatus::L1ViewStale => StartupAction::Refuse(RefuseReason::L1ViewStale), - DangerStatus::EstimatedBatchInDanger(batch_index) => { - StartupAction::Refuse(RefuseReason::EstimatedBatchInDanger { batch_index }) - } +/// The sole startup policy function. Terminal local facts are ranked before +/// phase progress, so no provider call or mutation can mask divergence or a +/// missing finalized state. +fn reduce_recovery(progress: RecoveryProgress, facts: RecoveryInspection) -> RecoveryDecision { + if let DangerStatus::CanonicalDivergence(nonce) = facts.danger { + return RecoveryDecision::Refuse(RecoveryRefusalReason::CanonicalDivergence { nonce }); + } + if !facts.has_finalized_snapshot { + return RecoveryDecision::Refuse(RecoveryRefusalReason::MissingFinalizedSnapshot); + } + + match progress { + RecoveryProgress::NeedInitialSync => RecoveryDecision::Act(RecoveryPhase::InitialSync), + RecoveryProgress::Flushed { + observed_safe_block, + } => RecoveryDecision::Act(RecoveryPhase::PostFlushSync { + required_safe_block: observed_safe_block, + }), + RecoveryProgress::PostFlushSynced { + required_safe_block, + } => match facts.current_safe_block { + None => RecoveryDecision::Refuse(RecoveryRefusalReason::MissingSafeHead), + Some(resynced_safe_block) if resynced_safe_block < required_safe_block => { + RecoveryDecision::Retry(RecoveryRetryReason::ResyncBehindFlushView { + resynced_safe_block, + flush_observed_safe_block: required_safe_block, + }) + } + Some(_) => RecoveryDecision::Act(RecoveryPhase::Cascade { + required_safe_block, + }), + }, + RecoveryProgress::Repaired => match facts.danger { + DangerStatus::Safe if facts.has_open_tip => RecoveryDecision::Admit, + // Unreachable in production — every repair phase ends with a + // valid open tip in its own transaction (the admission model + // encodes that postcondition). Kept so the `Repaired` and + // `Inspecting` arms stay structurally parallel and total. + DangerStatus::Safe => RecoveryDecision::Act(RecoveryPhase::EnsureOpenTip), + // Named, not a wildcard: a new `DangerStatus` variant must be + // classified here explicitly instead of silently defaulting to + // retry — the exact mistake this reducer exists to make + // compiler-visible. + status @ (DangerStatus::ClosedBatchInDanger(_) + | DangerStatus::TipInDanger(_) + | DangerStatus::L1ViewStale + | DangerStatus::EstimatedBatchInDanger(_)) => { + RecoveryDecision::Retry(RecoveryRetryReason::DangerPersists { status }) + } + DangerStatus::CanonicalDivergence(_) => { + unreachable!("terminal facts were reduced before progress") + } + }, + RecoveryProgress::Inspecting => match facts.danger { + DangerStatus::Safe if facts.has_open_tip => RecoveryDecision::Admit, + DangerStatus::Safe => RecoveryDecision::Act(RecoveryPhase::EnsureOpenTip), + DangerStatus::ClosedBatchInDanger(_) => RecoveryDecision::Act(RecoveryPhase::Flush), + DangerStatus::TipInDanger(expected_batch_index) => { + RecoveryDecision::Act(RecoveryPhase::RecoverTip { + expected_batch_index, + }) + } + DangerStatus::L1ViewStale => RecoveryDecision::Retry(RecoveryRetryReason::L1ViewStale), + DangerStatus::EstimatedBatchInDanger(batch_index) => { + RecoveryDecision::Retry(RecoveryRetryReason::EstimatedBatchInDanger { batch_index }) + } + DangerStatus::CanonicalDivergence(_) => { + unreachable!("terminal facts were reduced before progress") + } + }, } } -/// Run the full preemptive recovery procedure at startup. -/// -/// 1. Try to sync the safe head from L1. If L1 is unreachable, continue with -/// the persisted view; whether that view is fresh enough is decided by -/// `check_danger` in step 2 — a stale persisted view returns -/// `L1ViewStale` and step 3 refuses. -/// 2. Consult [`decide_startup_action`] to pick what to do. -/// 3. If the decision is `FlushAndCascade`: flush the mempool, re-sync, then -/// continue. If `Refuse`: bail out and let the orchestrator retry. -/// 4. Run the atomic recovery transaction (cascade stale batches if any, -/// always re-open the Tip if missing). -/// -/// Returns the list of invalidated batch indices (empty if no stale batches). -pub async fn run_preemptive_recovery( - db_path: &str, - input_reader: &mut InputReader, - l1_config: &L1Config, - protocol: &ProtocolTiming, -) -> Result, RecoveryError> { - // ── Step 1: Sync safe head (tolerate L1 failure) ─────────────── - // - // `sync_to_current_safe_head` goes through `append_safe_inputs`, which - // maintains `safe_accepted_batches` atomically with each advance. After - // a successful sync, the scheduler-frontier view is consistent with - // l1_safe_head for every downstream reader. - let l1_reachable = match input_reader.sync_to_current_safe_head().await { - Ok(()) => { - tracing::info!("L1 safe head synced"); - true - } - Err(e) => { - let InputReaderError::Provider(error) = e else { - return Err(RecoveryError::InputReader(e)); - }; - tracing::error!(error = %error, "L1 unreachable during startup safe-head sync"); - false - } - }; +trait RecoveryDriver { + fn inspect(&mut self) -> Result; - // ── Step 2: Read danger and decide action ───────────────────── - let danger = { - let mut storage = storage::Storage::open(db_path)?; - storage.check_danger(protocol, crate::runtime::clock::unix_now_ms())? - }; - let action = decide_startup_action(danger); - tracing::info!( - danger_status = danger.label(), - danger_batch_index = ?danger.batch_index(), - startup_action = action.label(), - l1_reachable, - danger_threshold = protocol.danger_threshold(), - max_wait_blocks = protocol.max_wait_blocks, - l1_read_stale_after_blocks = protocol.l1_read_stale_after_blocks, - "startup recovery decision" - ); - - // ── Step 3: Execute decision ─────────────────────────────────── - // - // The three non-Refuse paths split the recovery work: - // - // - `Proceed`: no DB writes. A `Proceed` decision means no batch is in - // danger and the persisted state is fine as-is. Closed batches past - // gold (if any) stay in their natural lifecycle. - // - // - `RecoverTip`: no flush. Only the open Tip crossed `danger_threshold`; - // it has no L1 slot to resolve, so it can be invalidated directly. - // - // - `FlushAndCascade`: flush resolves every wallet-nonce slot, then - // re-sync brings the gold frontier to its maximum extent. After that - // point, *everything past gold is doomed* (Silver-stale, - // Silver-poisoned, or Pending-killed — see `Storage::recover_post_flush` - // docs). Cascade unconditionally from the first non-gold. - let invalidated = match action { - StartupAction::Proceed => { - tracing::info!( - danger_status = danger.label(), - danger_batch_index = ?danger.batch_index(), - startup_action = action.label(), - "no danger zone detected — proceeding without recovery" - ); - // No DB writes here. A `Proceed` decision means no batch is in - // danger and the persisted state is fine as-is; closed batches past - // gold (if any) stay in their natural lifecycle. The tip-existence - // invariant — including opening the genesis Tip on a fresh DB — is - // established structurally by `Storage::ensure_open_tip` in - // `Workers::spawn`, after this returns and before the lane starts. - Vec::new() - } - StartupAction::RecoverTip { batch_index } => { - tracing::error!( - danger_status = danger.label(), - danger_batch_index = ?danger.batch_index(), - startup_action = action.label(), - tip_batch_index = batch_index, - danger_threshold = protocol.danger_threshold(), - "open Tip in danger zone — invalidating and opening fresh Tip (no flush)" - ); - let mut storage = storage::Storage::open(db_path)?; - storage.recover_aging_tip(protocol.danger_threshold())? - } - StartupAction::FlushAndCascade { batch_index } => { - tracing::error!( - danger_status = danger.label(), - danger_batch_index = ?danger.batch_index(), - startup_action = action.label(), - batch_index, - danger_threshold = protocol.danger_threshold(), - max_wait_blocks = protocol.max_wait_blocks, - "closed batch in danger zone — entering preemptive recovery (flush + cascade)" - ); - run_flush_and_cascade(db_path, input_reader, l1_config, protocol).await? - } - StartupAction::Refuse(reason) => { - tracing::error!( - danger_status = danger.label(), - danger_batch_index = ?danger.batch_index(), - startup_action = action.label(), - ?reason, - refuse_reason = reason.label(), - l1_reachable, - "startup refused: cannot recover safely" - ); - return Err(RecoveryError::Refuse(reason)); - } - }; + /// Perform one phase and return the progress it established. The + /// production driver derives it from the phase itself, so a + /// wrong-progress return is unrepresentable there. + async fn perform(&mut self, phase: RecoveryPhase) -> Result; - if invalidated.is_empty() { + fn admitted(&mut self) {} +} + +/// Drive one phase per inspection. There is intentionally no edge from a +/// completed phase directly to another phase or admission. +async fn drive_recovery(driver: &mut impl RecoveryDriver) -> Result<(), RecoveryError> { + let mut progress = RecoveryProgress::NeedInitialSync; + loop { + let facts = driver.inspect()?; + let decision = reduce_recovery(progress, facts); tracing::info!( - danger_status = danger.label(), - danger_batch_index = ?danger.batch_index(), - startup_action = action.label(), - invalidated_count = 0, - "startup recovery complete — no batches invalidated" + recovery_progress = ?progress, + danger_status = facts.danger.label(), + danger_batch_index = ?facts.danger.batch_index(), + recovery_decision = decision.label(), + "startup recovery reducer decision" ); + + match decision { + RecoveryDecision::Admit => { + driver.admitted(); + return Ok(()); + } + RecoveryDecision::Retry(reason) => return Err(RecoveryError::retry(reason)), + RecoveryDecision::Refuse(reason) => return Err(RecoveryError::refuse(reason)), + RecoveryDecision::Act(phase) => { + progress = driver.perform(phase).await?; + } + } + } +} + +fn log_repair(invalidated: &[u64]) { + if invalidated.is_empty() { + tracing::info!("startup recovery phase completed without invalidation"); } else { - // Successful self-heal: the system invalidated the doomed suffix and - // opened a recovery batch as designed. The upstream "danger detected" - // log already alerted the operator at error level; this completes - // that incident with a non-error outcome. tracing::warn!( - danger_status = danger.label(), - danger_batch_index = ?danger.batch_index(), - startup_action = action.label(), invalidated_count = invalidated.len(), batches = ?invalidated, - "startup recovery complete — batches invalidated and recovery batch opened" + "startup recovery invalidated the doomed suffix" ); } +} - Ok(invalidated) +struct ProductionRecoveryDriver<'a> { + db_path: &'a str, + input_reader: &'a mut InputReader, + l1_config: &'a L1Config, + protocol: &'a ProtocolTiming, } -/// Execute the flush-and-cascade phase: resolve every pending wallet-nonce -/// slot on L1, re-sync the safe head so the gold frontier reflects post-flush -/// state, then cascade-invalidate the doomed non-gold suffix and open a fresh -/// recovery Tip. -/// -/// The four steps form one logical phase — they have no meaning on their own -/// and the orchestrator only ever runs them as a unit. -async fn run_flush_and_cascade( +impl RecoveryDriver for ProductionRecoveryDriver<'_> { + fn inspect(&mut self) -> Result { + let mut storage = storage::Storage::open_writer(self.db_path).map_err(classify_open)?; + storage + .inspect_recovery(self.protocol, crate::clock::unix_now_ms()) + .map_err(classify_storage) + } + + async fn perform(&mut self, phase: RecoveryPhase) -> Result { + match phase { + RecoveryPhase::InitialSync => { + match self.input_reader.sync_to_current_safe_head().await { + Ok(()) => tracing::info!("L1 safe head synced"), + // Preserve warm boot: an unreachable provider counts as a + // completed refresh attempt, then persisted local facts + // decide whether serving is still honest. + Err(InputReaderError::Provider(error)) => tracing::warn!( + error = %error, + "L1 unreachable during initial startup sync; inspecting persisted view" + ), + Err(error) => return Err(classify_input_reader(error)), + } + Ok(RecoveryProgress::Inspecting) + } + RecoveryPhase::EnsureOpenTip => { + let mut storage = + storage::Storage::open_writer(self.db_path).map_err(classify_open)?; + storage + .ensure_open_tip_for_recovery(self.protocol, crate::clock::unix_now_ms()) + .map_err(classify_mutation)?; + log_repair(&[]); + Ok(RecoveryProgress::Repaired) + } + RecoveryPhase::RecoverTip { + expected_batch_index, + } => { + let mut storage = + storage::Storage::open_writer(self.db_path).map_err(classify_open)?; + let invalidated = storage + .recover_aging_tip_for_recovery( + expected_batch_index, + self.protocol, + crate::clock::unix_now_ms(), + ) + .map_err(classify_mutation)?; + log_repair(&invalidated); + Ok(RecoveryProgress::Repaired) + } + RecoveryPhase::Flush => { + let observed_safe_block = self.flush().await?; + Ok(RecoveryProgress::Flushed { + observed_safe_block, + }) + } + RecoveryPhase::PostFlushSync { + required_safe_block, + } => { + self.input_reader + .sync_to_current_safe_head() + .await + .map_err(classify_input_reader)?; + Ok(RecoveryProgress::PostFlushSynced { + required_safe_block, + }) + } + RecoveryPhase::Cascade { + required_safe_block, + } => { + let mut storage = + storage::Storage::open_writer(self.db_path).map_err(classify_open)?; + let invalidated = storage + .recover_post_flush_for_recovery( + required_safe_block, + self.protocol, + crate::clock::unix_now_ms(), + ) + .map_err(classify_mutation)?; + log_repair(&invalidated); + Ok(RecoveryProgress::Repaired) + } + } + } +} + +impl ProductionRecoveryDriver<'_> { + async fn flush(&mut self) -> Result { + use crate::l1::provider::VerifiedSignerProviderError; + + let provider = crate::l1::provider::create_verified_signer_provider( + &self.l1_config.eth_rpc_url, + self.l1_config.batch_submitter_private_key.expose_secret(), + self.l1_config.identity.chain_id, + self.l1_config.allow_insecure_rpc, + ) + .await + .map_err(|error| match error { + VerifiedSignerProviderError::ChainIdMismatch { rpc, expected } => { + RecoveryError::refuse(RecoveryFailure::ChainIdMismatch { rpc, expected }) + } + VerifiedSignerProviderError::ChainIdRpc(message) => { + RecoveryError::retry(RecoveryFailure::Provider(message)) + } + VerifiedSignerProviderError::Create(message) => { + RecoveryError::refuse(RecoveryFailure::Provider(message)) + } + })?; + + let watermark = { + let mut storage = storage::Storage::open_writer(self.db_path).map_err(classify_open)?; + storage.wallet_nonce_watermark().map_err(classify_storage)? + }; + MempoolFlusher::flush_to_safe( + provider, + self.l1_config.identity.batch_submitter_address, + self.protocol.seconds_per_block, + self.db_path, + watermark, + ) + .await + .map_err(classify_flush) + } +} + +/// Run the startup reducer through its first clean `Admit` decision. This +/// grants no runtime capability; fallible runtime preparation follows, then +/// [`admit_runtime`] invokes the same reducer once more over one consistent +/// fact set. +pub(crate) async fn run_startup_recovery( db_path: &str, input_reader: &mut InputReader, l1_config: &L1Config, protocol: &ProtocolTiming, -) -> Result, RecoveryError> { - // Keyed-write chain-id gate (review): the flush signs L1 no-op txs, so it - // must confirm the RPC still serves the pinned chain *immediately before - // signing*. The boot-time `validate_rpc_chain_id` and the reader's one-shot - // `verify_chain_id` are both stale by now (a load-balanced RPC could have - // failed over to another chain since), so neither is a sufficient backstop - // for a fresh keyed write. `create_verified_signer_provider` folds the check - // into the signer build so this path cannot skip it. A mismatch is terminal - // (operator misconfig); an RPC error is retryable (handled like `Provider`). - let flush_provider = crate::l1::provider::create_verified_signer_provider( - &l1_config.eth_rpc_url, - &l1_config.batch_submitter_private_key, - l1_config.chain_id, - l1_config.allow_insecure_rpc, - ) - .await - .map_err(|e| match e { - crate::l1::provider::VerifiedSignerProviderError::ChainIdMismatch { rpc, expected } => { - RecoveryError::ChainIdMismatch { rpc, expected } - } - other => RecoveryError::Provider(other.to_string()), - })?; - // The persisted watermark anchors the flush: every slot this deployment - // ever broadcast must resolve at safe depth, regardless of what the - // local node's pool remembers (review R1a / F1). - let watermark = { - let mut storage = storage::Storage::open(db_path)?; - storage.wallet_nonce_watermark()? - }; - let flush_observed_safe_block = MempoolFlusher::flush_to_safe( - flush_provider, - l1_config.batch_submitter_address, - protocol.seconds_per_block, +) -> Result<(), RecoveryError> { + let mut driver = ProductionRecoveryDriver { db_path, - watermark, - ) - .await?; - - // If this re-sync errors out, L1 has been flushed but the DB has NOT been - // cascaded — we exit with the InputReaderError and rely on the orchestrator - // to respawn. That's safe by design: - // - // - `flush_and_wait` is idempotent: on the next attempt it queries L1 for - // pending wallet-nonces, finds zero (the previous flush cleared them), - // and returns immediately. - // - `check_danger` re-decides on the post-flush state. Two cases: the - // danger persists (the restart re-enters this same path — flush is a - // no-op the second time), or the original danger resolved during the - // flush (e.g. the frontier batch landed gold), in which case the - // restart proceeds normally with any no-op'd Pending batch left valid — - // safe, since it simply resubmits at a fresh slot with no poisoned - // ancestor. - // - `recover_post_flush` is idempotent against the resulting DB state - // (verified by `after_post_recovery_crash_is_no_op` in `recovery_tests`). - // - // So a failure here just costs an extra orchestrator respawn; correctness - // is preserved. - // - // More importantly, it refuses to boot, during a recovery scenario, when - // we can't reach L1. - tracing::info!("re-syncing L1 safe head after flush"); - input_reader.sync_to_current_safe_head().await?; - - tracing::info!("running post-flush recovery (cascade non-gold suffix)"); - let mut storage = storage::Storage::open(db_path)?; - - // Coherence check (review F2): the cascade's precondition is that the - // gold frontier reflects at least the safe view the flusher observed - // resolution at. Behind a load-balanced RPC, the reader's re-sync can be - // served by a replica lagging the flusher's view — cascading then could - // invalidate a batch the scheduler actually accepted and reuse its - // nonce. Refuse instead; the orchestrator respawn retries with a - // fresher view (the flush is idempotent). - let resynced_safe_block = storage.current_safe_block()?.unwrap_or(0); - assert_resync_caught_up(resynced_safe_block, flush_observed_safe_block)?; - - Ok(storage.recover_post_flush(protocol.danger_threshold())?) + input_reader, + l1_config, + protocol, + }; + drive_recovery(&mut driver).await +} + +/// Reinvoke the same reducer after all fallible preparation, over one +/// transactionally consistent fact set. Anything except `Admit` drops the +/// prepared resources and restarts from a fresh boot; workers are never +/// launched from an aged decision. +/// +/// This consistent read *is* the linearization of the final admission +/// decision: the process lock excludes every other process, and no worker +/// is launched until after this decision — the launch step itself is +/// non-yielding — so no writer exists that could invalidate the facts +/// between this read and worker launch. +pub(crate) fn admit_runtime( + db_path: &str, + protocol: &ProtocolTiming, +) -> Result { + let mut storage = storage::Storage::open_writer(db_path).map_err(classify_open)?; + let facts = storage + .inspect_recovery(protocol, crate::clock::unix_now_ms()) + .map_err(classify_storage)?; + match reduce_recovery(RecoveryProgress::Inspecting, facts) { + RecoveryDecision::Admit => Ok(RuntimeAdmission { _private: () }), + RecoveryDecision::Retry(reason) => Err(RecoveryError::retry(reason)), + RecoveryDecision::Refuse(reason) => Err(RecoveryError::refuse(reason)), + RecoveryDecision::Act(phase) => Err(RecoveryError::retry( + RecoveryRetryReason::AdmissionChanged { + decision: phase.label(), + }, + )), + } +} + +fn classify_open(error: StorageOpenError) -> RecoveryError { + let persistent = storage::is_persistent_storage_open_error(&error); + let failure = RecoveryFailure::OpenStorage(error); + if persistent { + RecoveryError::refuse(failure) + } else { + RecoveryError::retry(failure) + } +} + +fn classify_storage(error: rusqlite::Error) -> RecoveryError { + let persistent = storage::is_persistent_storage_error(&error); + let failure = RecoveryFailure::Storage(error); + if persistent { + RecoveryError::refuse(failure) + } else { + RecoveryError::retry(failure) + } +} + +fn classify_input_reader(error: InputReaderError) -> RecoveryError { + match error { + error @ (InputReaderError::Provider(_) | InputReaderError::InconsistentL1Response(_)) => { + RecoveryError::retry(RecoveryFailure::InputReader(error)) + } + InputReaderError::OpenStorage(source) => classify_open(source), + InputReaderError::Storage(source) => classify_storage(source), + error @ (InputReaderError::ChainIdMismatch { .. } + | InputReaderError::Bootstrap(_) + | InputReaderError::StorageTaskPanicked { .. } + | InputReaderError::Join(_)) => RecoveryError::refuse(RecoveryFailure::InputReader(error)), + } +} + +fn classify_flush(error: FlushError) -> RecoveryError { + let terminal = error.is_terminal_invariant(); + let failure = RecoveryFailure::Flush(error); + if terminal { + RecoveryError::refuse(failure) + } else { + RecoveryError::retry(failure) + } +} + +fn classify_mutation(error: RecoveryMutationError) -> RecoveryError { + match error { + RecoveryMutationError::Storage(source) => classify_storage(source), + RecoveryMutationError::CanonicalDivergence { nonce } => { + RecoveryError::refuse(RecoveryRefusalReason::CanonicalDivergence { nonce }) + } + RecoveryMutationError::MissingFinalizedSnapshot => { + RecoveryError::refuse(RecoveryRefusalReason::MissingFinalizedSnapshot) + } + RecoveryMutationError::MissingSafeHead => { + RecoveryError::refuse(RecoveryRefusalReason::MissingSafeHead) + } + RecoveryMutationError::ResyncBehindFlushView { + resynced_safe_block, + flush_observed_safe_block, + } => RecoveryError::retry(RecoveryRetryReason::ResyncBehindFlushView { + resynced_safe_block, + flush_observed_safe_block, + }), + RecoveryMutationError::StaleDecision { actual, .. } => { + RecoveryError::retry(RecoveryRetryReason::StaleDecision { status: actual }) + } + } } #[cfg(test)] mod tests { + use std::collections::VecDeque; + use super::*; + fn sqlite_failure(code: rusqlite::ffi::ErrorCode, extended_code: i32) -> rusqlite::Error { + rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code, + extended_code, + }, + None, + ) + } + + fn assert_retry(error: RecoveryError) { + assert!( + matches!(error, RecoveryError::Retry(_)), + "expected retry, got {error:?}" + ); + } + + fn assert_refuse(error: RecoveryError) { + assert!( + matches!(error, RecoveryError::Refuse(_)), + "expected refusal, got {error:?}" + ); + } + + #[test] + fn error_classifiers_pin_retry_and_refuse_polarity() { + use crate::l1::watermark::WalletNonceWatermarkError; + + let busy = || sqlite_failure(rusqlite::ffi::ErrorCode::DatabaseBusy, 5); + let corrupt = || sqlite_failure(rusqlite::ffi::ErrorCode::NotADatabase, 26); + + assert_retry(classify_open(StorageOpenError::Sqlite(busy()))); + assert_refuse(classify_open(StorageOpenError::Sqlite(corrupt()))); + assert_retry(classify_storage(busy())); + assert_refuse(classify_storage(rusqlite::Error::QueryReturnedNoRows)); + + assert_retry(classify_input_reader(InputReaderError::Provider( + "offline".into(), + ))); + assert_refuse(classify_input_reader(InputReaderError::ChainIdMismatch { + rpc: 1, + expected: 2, + })); + + assert_retry(classify_flush(FlushError::Provider("offline".into()))); + assert_refuse(classify_flush(FlushError::Watermark( + WalletNonceWatermarkError::Storage(rusqlite::Error::QueryReturnedNoRows), + ))); + + assert_retry(classify_mutation(RecoveryMutationError::StaleDecision { + expected: DangerStatus::Safe, + actual: DangerStatus::L1ViewStale, + })); + assert_retry(classify_mutation( + RecoveryMutationError::ResyncBehindFlushView { + resynced_safe_block: 10, + flush_observed_safe_block: 11, + }, + )); + assert_refuse(classify_mutation( + RecoveryMutationError::CanonicalDivergence { nonce: 7 }, + )); + assert_refuse(classify_mutation( + RecoveryMutationError::MissingFinalizedSnapshot, + )); + assert_refuse(classify_mutation(RecoveryMutationError::MissingSafeHead)); + } + + fn facts(danger: DangerStatus) -> RecoveryInspection { + RecoveryInspection { + danger, + has_finalized_snapshot: true, + has_open_tip: true, + current_safe_block: Some(1_200), + } + } + + #[test] + fn divergence_dominates_every_progress_state() { + let terminal = facts(DangerStatus::CanonicalDivergence(7)); + for progress in [ + RecoveryProgress::NeedInitialSync, + RecoveryProgress::Inspecting, + RecoveryProgress::Flushed { + observed_safe_block: 1_201, + }, + RecoveryProgress::PostFlushSynced { + required_safe_block: 1_201, + }, + RecoveryProgress::Repaired, + ] { + assert_eq!( + reduce_recovery(progress, terminal), + RecoveryDecision::Refuse(RecoveryRefusalReason::CanonicalDivergence { nonce: 7 }) + ); + } + } + #[test] - fn proceed_on_safe() { + fn post_flush_lag_retries_before_cascade() { + let decision = reduce_recovery( + RecoveryProgress::PostFlushSynced { + required_safe_block: 1_201, + }, + facts(DangerStatus::ClosedBatchInDanger(0)), + ); assert_eq!( - decide_startup_action(DangerStatus::Safe), - StartupAction::Proceed + decision, + RecoveryDecision::Retry(RecoveryRetryReason::ResyncBehindFlushView { + resynced_safe_block: 1_200, + flush_observed_safe_block: 1_201, + }) ); } #[test] - fn refuse_on_canonical_divergence() { - // Terminal refusal — never `Proceed`, never a flush+cascade on top - // of a diverged frontier (review R2). The remedy is cockroach - // recovery, outside this dispatch entirely. + fn repaired_tip_with_surviving_clock_refusal_cannot_admit() { assert_eq!( - decide_startup_action(DangerStatus::CanonicalDivergence(7)), - StartupAction::Refuse(RefuseReason::CanonicalDivergence { nonce: 7 }) + reduce_recovery(RecoveryProgress::Repaired, facts(DangerStatus::L1ViewStale)), + RecoveryDecision::Retry(RecoveryRetryReason::DangerPersists { + status: DangerStatus::L1ViewStale + }) ); } - #[test] - fn flush_and_cascade_on_closed_batch_in_danger() { + struct ScriptedDriver { + inspections: VecDeque, + trace: Vec<&'static str>, + flush_observed_safe_block: u64, + inspection_attempts: usize, + fail_inspection_at: Option, + } + + impl ScriptedDriver { + fn new(inspections: impl IntoIterator) -> Self { + Self { + inspections: inspections.into_iter().collect(), + trace: Vec::new(), + flush_observed_safe_block: 1_200, + inspection_attempts: 0, + fail_inspection_at: None, + } + } + + fn fail_inspection_at(mut self, attempt: usize) -> Self { + self.fail_inspection_at = Some(attempt); + self + } + } + + impl RecoveryDriver for ScriptedDriver { + fn inspect(&mut self) -> Result { + self.trace.push("inspect"); + self.inspection_attempts += 1; + if self.fail_inspection_at == Some(self.inspection_attempts) { + return Err(RecoveryError::retry(RecoveryRetryReason::L1ViewStale)); + } + Ok(self + .inspections + .pop_front() + .expect("script provides one fact set per inspection")) + } + + async fn perform( + &mut self, + phase: RecoveryPhase, + ) -> Result { + Ok(match phase { + RecoveryPhase::InitialSync => { + self.trace.push("initial_sync"); + RecoveryProgress::Inspecting + } + RecoveryPhase::EnsureOpenTip => { + self.trace.push("ensure_tip"); + RecoveryProgress::Repaired + } + RecoveryPhase::RecoverTip { .. } => { + self.trace.push("recover_tip"); + RecoveryProgress::Repaired + } + RecoveryPhase::Flush => { + self.trace.push("flush"); + RecoveryProgress::Flushed { + observed_safe_block: self.flush_observed_safe_block, + } + } + RecoveryPhase::PostFlushSync { + required_safe_block, + } => { + self.trace.push("post_flush_sync"); + RecoveryProgress::PostFlushSynced { + required_safe_block, + } + } + RecoveryPhase::Cascade { .. } => { + self.trace.push("cascade"); + RecoveryProgress::Repaired + } + }) + } + + fn admitted(&mut self) { + self.trace.push("admit"); + } + } + + #[tokio::test] + async fn closed_recovery_runs_exactly_one_phase_per_inspection() { + let closed = facts(DangerStatus::ClosedBatchInDanger(0)); + let mut driver = + ScriptedDriver::new([closed, closed, closed, closed, facts(DangerStatus::Safe)]); + + drive_recovery(&mut driver).await.expect("admit"); assert_eq!( - decide_startup_action(DangerStatus::ClosedBatchInDanger(42)), - StartupAction::FlushAndCascade { batch_index: 42 } + driver.trace, + [ + "inspect", + "initial_sync", + "inspect", + "flush", + "inspect", + "post_flush_sync", + "inspect", + "cascade", + "inspect", + "admit", + ] ); } - #[test] - fn refuse_on_l1_view_stale() { + #[tokio::test] + async fn local_divergence_refuses_before_every_phase() { + let mut driver = ScriptedDriver::new([facts(DangerStatus::CanonicalDivergence(9))]); + let error = drive_recovery(&mut driver) + .await + .expect_err("divergence refuses"); + assert!(matches!(error, RecoveryError::Refuse(_))); + assert_eq!(driver.trace, ["inspect"]); + } + + #[tokio::test] + async fn sync_discovered_divergence_stops_before_cascade() { + let closed = facts(DangerStatus::ClosedBatchInDanger(0)); + let mut driver = ScriptedDriver::new([ + closed, + closed, + closed, + facts(DangerStatus::CanonicalDivergence(0)), + ]); + let error = drive_recovery(&mut driver) + .await + .expect_err("divergence discovered by sync refuses"); + assert!(matches!(error, RecoveryError::Refuse(_))); assert_eq!( - decide_startup_action(DangerStatus::L1ViewStale), - StartupAction::Refuse(RefuseReason::L1ViewStale) + driver.trace, + [ + "inspect", + "initial_sync", + "inspect", + "flush", + "inspect", + "post_flush_sync", + "inspect", + ] ); } - #[test] - fn refuse_on_estimated_batch_in_danger() { + #[tokio::test] + async fn tip_repair_reinspects_and_retries_on_surviving_clock_refusal() { + let tip = facts(DangerStatus::TipInDanger(0)); + let mut driver = ScriptedDriver::new([tip, tip, facts(DangerStatus::L1ViewStale)]); + + let error = drive_recovery(&mut driver) + .await + .expect_err("clock refusal must block admission after repair"); + assert!(matches!(error, RecoveryError::Retry(_))); assert_eq!( - decide_startup_action(DangerStatus::EstimatedBatchInDanger(7)), - StartupAction::Refuse(RefuseReason::EstimatedBatchInDanger { batch_index: 7 }) + driver.trace, + [ + "inspect", + "initial_sync", + "inspect", + "recover_tip", + "inspect", + ] ); } - #[test] - fn recover_tip_in_danger() { + #[tokio::test] + async fn reconstructed_controller_cannot_reuse_a_post_flush_sync_witness() { + let closed = facts(DangerStatus::ClosedBatchInDanger(0)); + let mut interrupted = ScriptedDriver::new([closed, closed, closed]).fail_inspection_at(4); + + let error = drive_recovery(&mut interrupted) + .await + .expect_err("the injected inspection boundary ends this controller"); + assert!(matches!(error, RecoveryError::Retry(_))); + assert_eq!( + interrupted.trace, + [ + "inspect", + "initial_sync", + "inspect", + "flush", + "inspect", + "post_flush_sync", + "inspect", + ], + "the first controller reached post-flush Sync before it was lost" + ); + + // Reconstructing the Rust controller is the restart boundary: its + // non-clone witnesses are gone. Even though durable facts still show + // the same closed danger, the new attempt must InitialSync and Flush; + // it cannot jump straight to Cascade using the previous attempt's + // PostFlushSync witness. + let mut restarted = + ScriptedDriver::new([closed, closed, closed, closed, facts(DangerStatus::Safe)]); + drive_recovery(&mut restarted) + .await + .expect("restart admits"); assert_eq!( - decide_startup_action(DangerStatus::TipInDanger(11)), - StartupAction::RecoverTip { batch_index: 11 } + restarted.trace, + [ + "inspect", + "initial_sync", + "inspect", + "flush", + "inspect", + "post_flush_sync", + "inspect", + "cascade", + "inspect", + "admit", + ] ); } + + fn admission_fixture( + name: &str, + has_finalized_snapshot: bool, + has_open_tip: bool, + ) -> (crate::storage::test_helpers::TestDb, ProtocolTiming) { + use crate::storage::test_helpers::{SENDER_A, default_protocol_timing, temp_db}; + + let db = temp_db(name); + let protocol = default_protocol_timing(); + let now_ms = crate::clock::unix_now_ms(); + let mut storage = + storage::Storage::initialize_for_command(&db.path, storage::LifecycleCommand::Setup) + .expect("initialize setup"); + storage + .append_safe_inputs_with_timestamp( + 0, + now_ms / 1_000, + &[], + SENDER_A, + &protocol, + storage::FrontierMode::Populate, + ) + .expect("seed fresh safe head"); + let prefix = db._dir.path().join("finalized"); + storage + .insert_initial_finalized_dump(&prefix, 0, 0, 0, 0) + .expect("seed finalized snapshot"); + if has_open_tip { + storage + .initialize_open_state(0, storage::SafeInputRange::empty_at(0)) + .expect("seed open Tip"); + } + storage.complete_setup().expect("complete setup"); + if !has_finalized_snapshot { + storage + .write(|tx| { + tx.execute("DELETE FROM finalized_snapshot", [])?; + Ok(()) + }) + .expect("simulate post-setup snapshot loss"); + } + drop(storage); + (db, protocol) + } + + #[test] + fn final_admission_refuses_new_divergence() { + let (db, protocol) = admission_fixture("admit-divergence", true, true); + let mut storage = storage::Storage::open_writer(&db.path).expect("open writer"); + crate::storage::test_helpers::record_canonical_divergence(&mut storage, 7, 0); + drop(storage); + + let error = + admit_runtime(&db.path, &protocol).expect_err("divergence must refuse final admission"); + assert!(matches!( + error, + RecoveryError::Refuse(failure) + if matches!( + *failure, + RecoveryFailure::PolicyRefusal( + RecoveryRefusalReason::CanonicalDivergence { nonce: 7 } + ) + ) + )); + } + + #[test] + fn final_admission_retries_when_tip_disappeared() { + let (db, protocol) = admission_fixture("admit-no-tip", true, false); + + let error = admit_runtime(&db.path, &protocol) + .expect_err("a missing Tip requires a fresh recovery attempt"); + assert!(matches!( + error, + RecoveryError::Retry(failure) + if matches!( + *failure, + RecoveryFailure::PolicyRetry( + RecoveryRetryReason::AdmissionChanged { + decision: "ensure_open_tip" + } + ) + ) + )); + } + + #[test] + fn final_admission_refuses_missing_snapshot() { + let (db, protocol) = admission_fixture("admit-no-snapshot", false, true); + + let error = admit_runtime(&db.path, &protocol) + .expect_err("a missing finalized snapshot must refuse final admission"); + assert!(matches!( + error, + RecoveryError::Refuse(failure) + if matches!( + *failure, + RecoveryFailure::PolicyRefusal( + RecoveryRefusalReason::MissingFinalizedSnapshot + ) + ) + )); + } } diff --git a/sequencer/src/runtime/clock.rs b/sequencer/src/runtime/clock.rs deleted file mode 100644 index a0874e86..00000000 --- a/sequencer/src/runtime/clock.rs +++ /dev/null @@ -1,19 +0,0 @@ -// (c) Cartesi and individual authors (see AUTHORS) -// SPDX-License-Identifier: Apache-2.0 (see LICENSE) - -//! Shared clock helper. -//! -//! Every callsite that needs "now in Unix-ms" goes through [`unix_now_ms`] so -//! the sequencer has a single place to swap in a test clock if needed. -//! `SystemTime::now()` pre-epoch is defended against via `unwrap_or_default()`. - -use std::time::SystemTime; - -/// Current wall-clock time as Unix-ms. Passed into -/// [`crate::storage::Storage::check_danger`] and friends. -pub fn unix_now_ms() -> u64 { - SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap_or_default() - .as_millis() as u64 -} diff --git a/sequencer/src/runtime/error.rs b/sequencer/src/runtime/error.rs deleted file mode 100644 index 058ac74e..00000000 --- a/sequencer/src/runtime/error.rs +++ /dev/null @@ -1,948 +0,0 @@ -// (c) Cartesi and individual authors (see AUTHORS) -// SPDX-License-Identifier: Apache-2.0 (see LICENSE) - -//! Runtime error taxonomy. Three groupings: -//! -//! - [`BootstrapError`] / [`IdentityError`]: everything that can go wrong -//! before runtime workers come up — config validation, deployment-identity -//! guards, startup recovery, initial DB open. -//! - [`WorkerExit`] + per-worker `*Exit`: how each runtime worker exited. -//! - [`RunError`]: the top-level error returned by `run()`, with generic -//! [`std::io::Error`] / [`rusqlite::Error`] catch-alls that are used widely -//! enough not to nest. - -use thiserror::Error; - -use crate::ingress::inclusion_lane::InclusionLaneError; -use crate::l1::fee_oracle::worker::FeeOracleError; -use crate::l1::reader::InputReaderError; -use crate::l1::submitter::{BatchPosterError, BatchSubmitterError}; -use crate::recovery::{DangerDetectorError, RecoveryError, RefuseReason}; -use crate::storage::{DangerStatus, DeploymentIdentity, StorageOpenError}; -use sequencer_core::protocol::ProtocolTimingError; - -// ── Top-level RunError ──────────────────────────────────────────────── - -/// Top-level runtime error. Grouped by phase: -/// -/// - `Bootstrap`: startup failures before runtime workers come up. -/// - `Worker`: one of the runtime workers exited (server, inclusion lane, -/// input reader, batch submitter, danger detector, fee oracle). -/// - `Io` / `Storage`: generic catch-alls used widely; not worth nesting. -#[derive(Debug, Error)] -pub enum RunError { - #[error("bootstrap failed: {0}")] - Bootstrap(#[from] BootstrapError), - #[error("worker exited: {0}")] - Worker(#[from] WorkerExit), - #[error(transparent)] - Io(#[from] std::io::Error), - #[error("storage operation failed: {0}")] - Storage(#[from] rusqlite::Error), - #[error("application bootstrap failed: {0}")] - AppBootstrap(#[from] sequencer_core::application::AppError), -} - -// ── R4 exit-code projection (WP10) ───────────────────────────────────── -// -// The orchestrator contract: a pure projection of `RunError` into a process -// exit code so the supervisor can tell "restart me" from "restarting is -// futile" without parsing logs. Lives here (one match), called by the harness -// so every app binary inherits it. The exit code is an ops hint, never -// protocol — authority over what the next boot does stays with startup's own -// `check_danger`. Reserved: 1 (unclassified), 2 (clap usage), 101 (panic). - -/// Restart with backoff; a recovery boot is expected next (it may take 15+ -/// min: flush + safe-finality wait). Startup probes must accommodate it. -pub const EXIT_RESTART_EXPECT_RECOVERY: u8 = 10; -/// Restart with backoff; a transient refusal that self-heals when the L1 view -/// freshens. Alert only if it persists. -pub const EXIT_RESTART_TRANSIENT: u8 = 20; -/// Terminal — do not restart; page an operator. The state cannot self-heal. -pub const EXIT_TERMINAL: u8 = 30; -/// Sticky — do **not** auto-restart-loop; an operator must run -/// `setup --recovery`. The recovery sibling of [`EXIT_RESTART_EXPECT_RECOVERY`] -/// (10), but *operator-initiated*: 10 means "restart and `run()` auto-recovers"; -/// 40 means "a previous instance left work past the checkpoint, and only an -/// explicit `setup --recovery` (PR5) can resolve it" — a plain restart of -/// `setup` would re-detect and re-refuse forever. Distinct from terminal (30) -/// in that a known recovery command *does* fix it. -pub const EXIT_SETUP_NEEDS_RECOVERY: u8 = 40; -/// Unclassified failure (worker crash, provider error). Restart with backoff. -pub const EXIT_UNCLASSIFIED: u8 = 1; - -impl RunError { - /// Project this error onto the R4 exit-code contract. Clean shutdown - /// (exit 0) is handled by the caller — a `RunError` is always a failure. - pub fn exit_code(&self) -> u8 { - match self { - RunError::Worker(WorkerExit::DangerDetector(DangerDetectorExit::DangerDetected { - status, - })) => danger_exit_code(status), - // A wrong-chain RPC caught by the reader after boot (the warm-boot - // deferral's backstop) is terminal, like the boot-time - // `BootstrapError::ChainIdMismatch` — a misconfig the operator must - // fix; a blind restart re-hits the same wrong chain. - RunError::Worker(WorkerExit::InputReader(InputReaderExit::Source( - InputReaderError::ChainIdMismatch { .. }, - ))) => EXIT_TERMINAL, - // Same for the submitter's pre-send chain-id gate: a wrong-chain RPC - // is an operator misconfig, terminal rather than a restart loop that - // keeps refusing to sign onto it. - RunError::Worker(WorkerExit::BatchSubmitter(BatchSubmitterExit::Source( - BatchSubmitterError::Poster(BatchPosterError::ChainIdMismatch { .. }), - ))) => EXIT_TERMINAL, - // Fee-oracle arithmetic that cannot encode without undercharging, or - // a pool/config misconfig discovered at runtime, is terminal — page - // an operator rather than restart-loop. - RunError::Worker(WorkerExit::FeeOracle(FeeOracleExit::Source( - FeeOracleError::FatalMath(_) | FeeOracleError::Misconfig(_), - ))) => EXIT_TERMINAL, - RunError::Bootstrap(b) => bootstrap_exit_code(b), - // Worker crashes, provider errors, IO/storage/app catch-alls. - RunError::Worker(_) - | RunError::Io(_) - | RunError::Storage(_) - | RunError::AppBootstrap(_) => EXIT_UNCLASSIFIED, - } - } -} - -fn danger_exit_code(status: &DangerStatus) -> u8 { - match status { - // A doomed closed batch / aging Tip: the next boot legitimately runs - // a slow recovery (flush + cascade). - DangerStatus::ClosedBatchInDanger(_) | DangerStatus::TipInDanger(_) => { - EXIT_RESTART_EXPECT_RECOVERY - } - // View-dependent refusals that self-heal once the provider recovers. - DangerStatus::L1ViewStale | DangerStatus::EstimatedBatchInDanger(_) => { - EXIT_RESTART_TRANSIENT - } - // The only genuinely terminal danger: canonical divergence (R2). - DangerStatus::CanonicalDivergence(_) => EXIT_TERMINAL, - // `Safe` is never a detector exit. If it ever reaches here the danger - // classification is self-contradicting — page an operator (EXIT_TERMINAL) - // rather than silently restart-loop with backoff (EXIT_UNCLASSIFIED). - DangerStatus::Safe => { - debug_assert!(false, "danger_exit_code called with DangerStatus::Safe"); - EXIT_TERMINAL - } - } -} - -fn bootstrap_exit_code(err: &BootstrapError) -> u8 { - match err { - // Transient: self-heal when the L1 view / provider recovers. - BootstrapError::ChainIdRpc { .. } - | BootstrapError::Identity(IdentityError::FirstBootRequiresL1) - | BootstrapError::DetectionNonceRead { .. } - | BootstrapError::Flush(_) => EXIT_RESTART_TRANSIENT, - BootstrapError::Recovery(RecoveryError::Refuse( - RefuseReason::L1ViewStale | RefuseReason::EstimatedBatchInDanger { .. }, - )) => EXIT_RESTART_TRANSIENT, - // A flush failure or a re-sync lagging the flush view during startup - // recovery is transient — the respawn retries with a fresher view. - // (Same class the `flush-mempool` subcommand gives a `FlushError`.) - BootstrapError::Recovery( - RecoveryError::Flush(_) | RecoveryError::ResyncBehindFlushView { .. }, - ) => EXIT_RESTART_TRANSIENT, - - // Terminal: needs an operator (wrong config, divergence, or a DB that - // was never set up). - BootstrapError::ChainIdMismatch { .. } - | BootstrapError::InvalidProtocolTiming(_) - | BootstrapError::FeeOracleMisconfig { .. } - | BootstrapError::SetupNotComplete - | BootstrapError::CheckpointBeforeAppDeployment { .. } - | BootstrapError::SetupRecovery(_) - | BootstrapError::Identity(IdentityError::Mismatch { .. } | IdentityError::OrphanedState) => { - EXIT_TERMINAL - } - - // Sticky setup refusal: a previous instance left work past the - // checkpoint. Distinct from the auto-recovery class (10) — a plain - // restart re-refuses; only `setup --recovery` (PR5) resolves it. - BootstrapError::SetupRefuse(_) => EXIT_SETUP_NEEDS_RECOVERY, - BootstrapError::Recovery(RecoveryError::Refuse(RefuseReason::CanonicalDivergence { - .. - })) => EXIT_TERMINAL, - // A wrong-chain RPC caught by the reader *during* a startup-recovery sync - // is terminal, like the boot-time and worker-path ChainIdMismatch arms — - // an operator misconfig a blind restart re-hits (without this it would - // fall to the catch-all below and loop on the wrong chain). - BootstrapError::Recovery(RecoveryError::InputReader( - InputReaderError::ChainIdMismatch { .. }, - )) => EXIT_TERMINAL, - // Same class for the preemptive-recovery flush's pre-signing chain-id - // gate: a mismatch is an operator misconfig, terminal rather than a - // restart loop on the wrong chain. (An RPC *error* during the check - // surfaces as `RecoveryError::Provider` and falls to the retry arm.) - BootstrapError::Recovery(RecoveryError::ChainIdMismatch { .. }) => EXIT_TERMINAL, - - // Other recovery / storage-open failures: unclassified, retry. - BootstrapError::Recovery(_) - | BootstrapError::OpenStorage(_) - | BootstrapError::FeeOracleTransient { .. } => EXIT_UNCLASSIFIED, - } -} - -// ── Bootstrap-phase errors ───────────────────────────────────────────── - -/// Anything that can go wrong before runtime workers start: config validation, -/// deployment-identity guards, startup recovery, initial DB open. -#[derive(Debug, Error)] -pub enum BootstrapError { - #[error(transparent)] - OpenStorage(#[from] StorageOpenError), - #[error("RPC chain ID {rpc} does not match the expected chain ID {config}")] - ChainIdMismatch { rpc: u64, config: u64 }, - /// `eth_chainId` failed on a reachable RPC. We treat this as fatal - /// rather than warn-and-continue: proceeding with an unverified chain id - /// would pin a possibly-wrong deployment identity and poison subsequent - /// L1-unreachable boots, in addition to issuing soft confirmations - /// against the wrong chain's state. Operator should retry. - #[error("could not query chain ID from RPC: {message}")] - ChainIdRpc { message: String }, - /// Protocol-level config (`preemptive_margin_blocks` vs `max_wait_blocks`, - /// `l1_read_stale_after_blocks` vs `danger_threshold`) failed validation. - /// See [`ProtocolTimingError`]. - #[error(transparent)] - InvalidProtocolTiming(#[from] ProtocolTimingError), - /// Setup-pinned fee oracle configuration or validation is invalid. - #[error("fee oracle misconfiguration: {message}")] - FeeOracleMisconfig { message: String }, - /// A live quote/transport failed while bootstrapping. It may self-heal. - #[error("fee oracle bootstrap transient failure: {message}")] - FeeOracleTransient { message: String }, - /// Startup recovery (or refusal) failed before runtime workers started. - #[error(transparent)] - Recovery(#[from] RecoveryError), - /// Deployment-identity guards — see [`IdentityError`]. - #[error(transparent)] - Identity(#[from] IdentityError), - /// `run` (or `flush-mempool`) was invoked against a DB where `setup` - /// has not completed — the `setup_complete` marker is absent (setup - /// never ran, or crashed midway), or its outputs are incomplete. The - /// operator must run `setup` first; restarting `run` cannot self-heal. - #[error("setup has not completed for this data dir — run `setup` first")] - SetupNotComplete, - /// The `flush-mempool` subcommand's flush failed (provider/transport). - #[error("mempool flush failed: {0}")] - Flush(#[from] crate::recovery::FlushError), - /// `setup`'s detection gate could not read the batch-submitter wallet - /// nonce from L1 (the one live RPC the gate makes). Transient — the - /// operator retries once the provider recovers; the prior sync already - /// proved L1 reachable, so this is a hiccup, not a misconfig. - #[error("setup detection: could not read submitter nonce from RPC: {message}")] - DetectionNonceRead { message: String }, - /// `setup`'s read-only detection gate found a previous instance left work - /// this checkpoint cannot account for. Sticky: only `setup --recovery` - /// (PR5) resolves it — a plain `setup` restart re-detects and re-refuses. - #[error(transparent)] - SetupRefuse(#[from] SetupRefuse), - /// `setup --checkpoint-block` predates the application's deployment block: - /// a promotion cannot have landed before the application contract existed. - /// Operator misconfig; restarting cannot self-heal. - #[error( - "checkpoint block {checkpoint_block} predates the application \ - deployment block {app_deployment_block}" - )] - CheckpointBeforeAppDeployment { - checkpoint_block: u64, - app_deployment_block: u64, - }, - /// `setup --recovery` failed in a way only the operator can fix (bad config, - /// a checkpoint that can't be loaded or doesn't fit the chain, or a DB that - /// is already set up). Terminal — see [`SetupRecoveryError`]. - #[error(transparent)] - SetupRecovery(#[from] SetupRecoveryError), -} - -/// Terminal failures of the `setup --recovery` procedure — the ones -/// an operator must resolve (the flush and the post-flush re-sync reuse the -/// transient [`RecoveryError`] paths instead). All map to [`EXIT_TERMINAL`]: -/// a plain restart re-runs the same bad inputs and re-fails identically. -#[derive(Debug, Error)] -pub enum SetupRecoveryError { - /// Cross-field config validation failed (recovery missing its dump dir / - /// checkpoint block / key, or a plain `setup` carrying recovery-only args). - /// See [`crate::runtime::config::SetupConfig::validate`]. - #[error("invalid recovery configuration: {message}")] - InvalidConfig { message: String }, - /// `setup --recovery` was invoked against a DB that is already set up. - /// Recovery is a strict one-shot on a freshly-wiped DB (wipe and re-run with - /// `--recovery`); re-pointing a live deployment at a different checkpoint - /// would strand its existing state. - #[error( - "`setup --recovery` requires a freshly-wiped data dir, but this one is \ - already set up — wipe it and re-run" - )] - AlreadySetUp, - /// The checkpoint dump could not be loaded (missing/corrupt `info.toml`, or - /// the app's `from_dump` failed). Operator must supply a valid **sequencer** - /// dump dir (`info.toml` + `state/`), not a watchdog CM checkpoint. - #[error("failed to load checkpoint dump at {path}: {message}")] - CheckpointLoad { path: String, message: String }, - /// The checkpoint's last-executed safe block `A` is not strictly before the - /// checkpoint block `B`. The fold reconstructs the `(A, B]` fridge, so - /// `A < B` must hold — otherwise the checkpoint dump and - /// `--checkpoint-block` describe inconsistent points. - #[error( - "checkpoint last-executed safe block {executed_safe_block} (A) is not \ - before checkpoint block {checkpoint_block} (B)" - )] - CheckpointNotBeforeBlock { - executed_safe_block: u64, - checkpoint_block: u64, - }, - /// A re-run of `setup --recovery` found a root tip from a *prior* (crashed - /// before the `setup_complete` marker) attempt whose nonce differs from this - /// attempt's resume nonce — a different checkpoint, or the same one after the - /// post-flush head `C` advanced. The half-recovered DB cannot be resumed - /// onto a tree rooted at the old nonce (the anchor would move but the - /// existing root tip would not, silently breaking I16). Wipe the data dir - /// and re-run. - #[error( - "partial recovery: existing root tip carries nonce {existing_root_nonce}, \ - but this attempt resumes at {requested_nonce} — wipe the data dir and re-run" - )] - PartialRecoveryMismatch { - existing_root_nonce: u64, - requested_nonce: u64, - }, - /// A re-run of `setup --recovery` found a root tip carrying *this* attempt's - /// resume nonce but **no finalized snapshot** — a prior attempt that crashed - /// between opening the root tip and writing the snapshot. It cannot be - /// resumed safely: a re-sync may have advanced `C` with new direct inputs - /// (which leave `N'` unchanged) that resuming would leave unsequenced, so the - /// snapshot cursor would lag the folded `S'` and `run` would drain+execute - /// them a second time (divergence). Wipe the data dir and re-run (the - /// one-shot recovery model). - #[error( - "partial recovery: root tip at nonce {root_nonce} exists with no finalized \ - snapshot (crashed mid-fill) — wipe the data dir and re-run" - )] - PartialRecoveryIncomplete { root_nonce: u64 }, - /// `setup --recovery` found a finalized snapshot but **no root tip**. A - /// completed cockroach fill always has both (the tip is opened in step 2, - /// before the snapshot in step 4), so this is residue from a *different* - /// deployment mode left in the data dir — a plain `setup` that registered the - /// genesis finalized snapshot and crashed before its `setup_complete` marker. - /// Folding `(S', N')` and then silently keeping the old snapshot would mark - /// setup complete over the genesis state instead of the recovered state. Wipe - /// the data dir and re-run `setup --recovery`. - #[error( - "setup --recovery found a finalized snapshot (block {existing_finalized_block}) \ - with no root tip — residue from an incomplete plain `setup`; wipe the data \ - dir and re-run" - )] - RecoveryOverResidualSnapshot { existing_finalized_block: u64 }, - /// A plain (non-recovery) `setup` found a non-zero batch-tree anchor — - /// residue from a `setup --recovery` that crashed before its marker. Booting - /// a genesis deployment over it would root the tree at the recovery nonce - /// instead of 0. Wipe the data dir, then run plain `setup` or re-run - /// `setup --recovery`. - #[error( - "plain setup found batch-tree anchor {anchor} (≠ 0) — leftover from an \ - incomplete `setup --recovery`; wipe the data dir and re-run" - )] - GenesisOverRecoveryResidue { anchor: u64 }, -} - -/// `setup`'s read-only detection gate: the reasons a -/// fresh `setup` refuses because a *previous* instance left work past the -/// checkpoint. The remedy is `setup --recovery` (PR5), which flushes/folds the -/// outstanding batches; a plain `setup` restart re-detects and re-refuses -/// (hence [`EXIT_SETUP_NEEDS_RECOVERY`], not the auto-recovery class 10). -/// -/// Both variants carry diagnostic fields for the refusal log line, mirroring -/// the danger-detector [`RefuseReason`] precedent. -#[derive(Debug, Error)] -pub enum SetupRefuse { - /// Step 1: the batch-submitter wallet nonce is not settled - /// (`pending > safe`) on the local provider — a previous instance left - /// pending or mined-but-unsafe batch txs. Local-view only (review F1): a - /// zombie tx dropped from this provider's pool but alive elsewhere evades - /// this check; bounded at runtime by the content-identity check. - #[error( - "batch-submitter wallet nonce not settled (pending {pending} > safe \ - {safe}) — a previous instance left in-flight batch txs; run \ - `setup --recovery`" - )] - WalletNonceUnsettled { pending: u64, safe: u64 }, - /// Step 2: a batch-submitter tx exists in `(checkpoint_block, safe]` — a - /// previous instance already wrote batches past this checkpoint, so a - /// genesis-style bootstrap would silently diverge from canonical state. - #[error( - "batch-submitter input found at block {found_block} past checkpoint \ - block {checkpoint_block} (safe_input_index {safe_input_index}) — run \ - `setup --recovery`" - )] - BatchPastCheckpoint { - checkpoint_block: u64, - found_block: u64, - safe_input_index: u64, - }, -} - -/// Deployment-identity failure modes. The sequencer pins itself to a specific -/// (chain_id, app_address, input_box_address, app_deployment_block, -/// batch_submitter_address) tuple on first successful boot, then refuses to -/// run under a different identity to prevent silently associating state from -/// one deployment with another. -#[derive(Debug, Error)] -pub enum IdentityError { - /// L1 unreachable AND no cached identity in the DB. We need at least one - /// (live L1 query OR a prior boot's pinned identity) to safely bind this - /// sequencer to a deployment. Operator: bring up L1 and retry. - #[error("first boot requires L1: no cached deployment identity and L1 is unreachable")] - FirstBootRequiresL1, - /// The DB has persisted state but no pinned identity. Binding the current - /// config now would silently inherit an unknown deployment's data. - /// Operator: confirm provenance or wipe the DB. - #[error("orphaned state: DB has persisted state but no deployment identity to claim it")] - OrphanedState, - /// The pinned identity doesn't match the current config. - /// - /// `stored` and `expected` are boxed so the enum stays small — without - /// boxing this variant alone would push `RunError`'s stack footprint past - /// 184 bytes, which clippy's `result_large_err` flags (and which inflates - /// every `Result<_, RunError>` in the codebase, even successful returns). - /// The heap allocation is paid only on the error path, which is cold. - #[error("deployment identity mismatch ({fields}); stored={stored:?}; expected={expected:?}")] - Mismatch { - fields: String, - stored: Box, - expected: Box, - }, -} - -// ── Worker exits ─────────────────────────────────────────────────────── - -/// Which runtime worker exited, and why. -#[derive(Debug, Error)] -pub enum WorkerExit { - #[error("server: {0}")] - Server(#[from] ServerExit), - #[error("inclusion lane: {0}")] - Lane(#[from] LaneExit), - #[error("input reader: {0}")] - InputReader(#[from] InputReaderExit), - #[error("batch submitter: {0}")] - BatchSubmitter(#[from] BatchSubmitterExit), - #[error("danger detector: {0}")] - DangerDetector(#[from] DangerDetectorExit), - #[error("fee oracle: {0}")] - FeeOracle(#[from] FeeOracleExit), -} - -/// Generic worker exit shape: stopped without signal / errored / failed to join. -#[derive(Debug, Error)] -pub enum ServerExit { - #[error("stopped unexpectedly")] - StoppedUnexpectedly, - #[error("io error: {0}")] - Source(std::io::Error), - #[error("join error: {0}")] - Join(tokio::task::JoinError), -} - -#[derive(Debug, Error)] -pub enum LaneExit { - #[error("stopped unexpectedly")] - StoppedUnexpectedly, - #[error("{0}")] - Source(InclusionLaneError), - #[error("join error: {0}")] - Join(tokio::task::JoinError), -} - -#[derive(Debug, Error)] -pub enum InputReaderExit { - #[error("stopped unexpectedly")] - StoppedUnexpectedly, - #[error("{0}")] - Source(InputReaderError), - #[error("join error: {0}")] - Join(tokio::task::JoinError), -} - -#[derive(Debug, Error)] -pub enum BatchSubmitterExit { - #[error("stopped unexpectedly")] - StoppedUnexpectedly, - #[error("{0}")] - Source(BatchSubmitterError), - #[error("join error: {0}")] - Join(tokio::task::JoinError), -} - -/// Detector has an extra variant for the deliberate `RecoveryRequired` trip: -/// not an error per se, but causes the runtime to exit so the orchestrator -/// can respawn into startup recovery. -#[derive(Debug, Error)] -pub enum DangerDetectorExit { - #[error("stopped unexpectedly")] - StoppedUnexpectedly, - #[error("{0}")] - Source(DangerDetectorError), - #[error("join error: {0}")] - Join(tokio::task::JoinError), - #[error("danger detected ({status:?}) — stopping for startup recovery")] - DangerDetected { status: DangerStatus }, -} - -#[derive(Debug, Error)] -pub enum FeeOracleExit { - #[error("stopped unexpectedly")] - StoppedUnexpectedly, - #[error("{0}")] - Source(FeeOracleError), - #[error("join error: {0}")] - Join(tokio::task::JoinError), -} - -// ── Shutdown-time constructors ──────────────────────────────────────── -// -// Used during orderly shutdown (runtime-wide shutdown was already -// requested). `Ok(())` is the expected "drained cleanly" outcome and -// returns `Ok(())`; everything else surfaces as the matching error variant. -// Distinct from the select-arm `From` impls, where `Ok(())` means the worker -// stopped *before* shutdown was triggered (`StoppedUnexpectedly`). - -impl ServerExit { - pub fn from_shutdown( - result: Result, tokio::task::JoinError>, - ) -> Result<(), Self> { - match result { - Ok(Ok(())) => Ok(()), - Ok(Err(source)) => Err(Self::Source(source)), - Err(source) => Err(Self::Join(source)), - } - } -} - -impl LaneExit { - pub fn from_shutdown( - result: Result, tokio::task::JoinError>, - ) -> Result<(), Self> { - match result { - Ok(Ok(())) => Ok(()), - Ok(Err(source)) => Err(Self::Source(source)), - Err(source) => Err(Self::Join(source)), - } - } -} - -impl InputReaderExit { - pub fn from_shutdown( - result: Result, tokio::task::JoinError>, - ) -> Result<(), Self> { - match result { - Ok(Ok(())) => Ok(()), - Ok(Err(source)) => Err(Self::Source(source)), - Err(source) => Err(Self::Join(source)), - } - } -} - -impl BatchSubmitterExit { - pub fn from_shutdown( - result: Result< - Result, - tokio::task::JoinError, - >, - ) -> Result<(), Self> { - match result { - Ok(Ok(crate::l1::submitter::SubmitterExit::Shutdown)) => Ok(()), - Ok(Err(source)) => Err(Self::Source(source)), - Err(source) => Err(Self::Join(source)), - } - } -} - -impl DangerDetectorExit { - pub fn from_shutdown( - result: Result< - Result, - tokio::task::JoinError, - >, - ) -> Result<(), Self> { - match result { - Ok(Ok(crate::recovery::DetectorExit::Shutdown)) => Ok(()), - Ok(Ok(crate::recovery::DetectorExit::RecoveryRequired { status })) => { - Err(Self::DangerDetected { status }) - } - Ok(Err(source)) => Err(Self::Source(source)), - Err(source) => Err(Self::Join(source)), - } - } -} - -impl FeeOracleExit { - pub fn from_shutdown( - result: Result, tokio::task::JoinError>, - ) -> Result<(), Self> { - match result { - Ok(Ok(())) => Ok(()), - Ok(Err(source)) => Err(Self::Source(source)), - Err(source) => Err(Self::Join(source)), - } - } -} - -// ── Chained `From` impls so `?` works at the top-level RunError ──────── -// -// thiserror's `#[from]` is one-level; nested propagation needs manual -// impls. Each leaf error type that can bubble up through `?` in `run()` -// gets a direct From for RunError. - -impl From for RunError { - fn from(e: StorageOpenError) -> Self { - RunError::Bootstrap(e.into()) - } -} - -impl From for RunError { - fn from(e: ProtocolTimingError) -> Self { - RunError::Bootstrap(e.into()) - } -} - -impl From for RunError { - fn from(e: RecoveryError) -> Self { - RunError::Bootstrap(e.into()) - } -} - -impl From for RunError { - fn from(e: IdentityError) -> Self { - RunError::Bootstrap(e.into()) - } -} - -impl From for RunError { - fn from(e: crate::recovery::FlushError) -> Self { - RunError::Bootstrap(BootstrapError::Flush(e)) - } -} - -impl From for RunError { - fn from(e: SetupRefuse) -> Self { - RunError::Bootstrap(BootstrapError::SetupRefuse(e)) - } -} - -impl From for RunError { - fn from(e: SetupRecoveryError) -> Self { - RunError::Bootstrap(BootstrapError::SetupRecovery(e)) - } -} - -impl From for RunError { - fn from(error: FeeOracleError) -> Self { - match error { - FeeOracleError::OpenStorage(error) => RunError::from(error), - FeeOracleError::Transient(message) => { - RunError::Bootstrap(BootstrapError::FeeOracleTransient { message }) - } - error => RunError::Bootstrap(BootstrapError::FeeOracleMisconfig { - message: error.to_string(), - }), - } - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::l1::fee_oracle::worker::FeeOracleError; - use crate::recovery::{FlushError, RecoveryError}; - use crate::storage::DeploymentIdentity; - use sequencer_core::protocol::ProtocolTimingError; - - fn danger(status: DangerStatus) -> RunError { - RunError::Worker(WorkerExit::DangerDetector( - DangerDetectorExit::DangerDetected { status }, - )) - } - - fn dummy_identity() -> DeploymentIdentity { - use alloy_primitives::Address; - DeploymentIdentity { - chain_id: 1, - app_address: Address::repeat_byte(0x11), - input_box_address: Address::repeat_byte(0x22), - app_deployment_block: 0, - batch_submitter_address: Address::repeat_byte(0x33), - fee_oracle: crate::storage::FeeOracleIdentity::Fixed { log_gas_price: 0 }, - } - } - - #[test] - fn r4_class_10_expect_recovery_boot() { - assert_eq!( - danger(DangerStatus::ClosedBatchInDanger(0)).exit_code(), - EXIT_RESTART_EXPECT_RECOVERY - ); - assert_eq!( - danger(DangerStatus::TipInDanger(3)).exit_code(), - EXIT_RESTART_EXPECT_RECOVERY - ); - } - - #[test] - fn r4_class_20_transient_refusal() { - assert_eq!( - danger(DangerStatus::L1ViewStale).exit_code(), - EXIT_RESTART_TRANSIENT - ); - assert_eq!( - danger(DangerStatus::EstimatedBatchInDanger(2)).exit_code(), - EXIT_RESTART_TRANSIENT - ); - assert_eq!( - RunError::Bootstrap(BootstrapError::Recovery(RecoveryError::Refuse( - RefuseReason::L1ViewStale - ))) - .exit_code(), - EXIT_RESTART_TRANSIENT - ); - assert_eq!( - RunError::Bootstrap(BootstrapError::Identity(IdentityError::FirstBootRequiresL1)) - .exit_code(), - EXIT_RESTART_TRANSIENT - ); - assert_eq!( - RunError::Bootstrap(BootstrapError::ChainIdRpc { - message: "x".into() - }) - .exit_code(), - EXIT_RESTART_TRANSIENT - ); - assert_eq!( - RunError::from(FlushError::Provider("x".into())).exit_code(), - EXIT_RESTART_TRANSIENT - ); - // A flush failure surfaced via startup recovery must land in the same - // class as the flush-mempool subcommand's FlushError (review M2). - assert_eq!( - RunError::Bootstrap(BootstrapError::Recovery(RecoveryError::Flush( - FlushError::Provider("x".into()) - ))) - .exit_code(), - EXIT_RESTART_TRANSIENT - ); - assert_eq!( - RunError::Bootstrap(BootstrapError::Recovery( - RecoveryError::ResyncBehindFlushView { - resynced_safe_block: 1, - flush_observed_safe_block: 2, - } - )) - .exit_code(), - EXIT_RESTART_TRANSIENT - ); - } - - #[test] - fn r4_class_30_terminal_operator_required() { - assert_eq!( - danger(DangerStatus::CanonicalDivergence(0)).exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::Bootstrap(BootstrapError::Recovery(RecoveryError::Refuse( - RefuseReason::CanonicalDivergence { nonce: 0 } - ))) - .exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::Bootstrap(BootstrapError::SetupNotComplete).exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::Bootstrap(BootstrapError::ChainIdMismatch { rpc: 1, config: 2 }).exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::Bootstrap(BootstrapError::InvalidProtocolTiming( - ProtocolTimingError::MarginNotLessThanMaxWait { - margin: 1200, - max_wait: 1200 - } - )) - .exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::Bootstrap(BootstrapError::Identity(IdentityError::OrphanedState)).exit_code(), - EXIT_TERMINAL - ); - // `setup --recovery` operator-fixable failures are terminal (a restart - // re-runs the same bad inputs). - assert_eq!( - RunError::from(SetupRecoveryError::AlreadySetUp).exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::Bootstrap(BootstrapError::Identity(IdentityError::Mismatch { - fields: "chain_id".into(), - stored: Box::new(dummy_identity()), - expected: Box::new(dummy_identity()), - })) - .exit_code(), - EXIT_TERMINAL - ); - // Partial-recovery residue (B1/B2): operator must wipe — terminal. - assert_eq!( - RunError::from(SetupRecoveryError::PartialRecoveryMismatch { - existing_root_nonce: 3, - requested_nonce: 5, - }) - .exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::from(SetupRecoveryError::GenesisOverRecoveryResidue { anchor: 7 }) - .exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::from(SetupRecoveryError::PartialRecoveryIncomplete { root_nonce: 3 }) - .exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::from(SetupRecoveryError::RecoveryOverResidualSnapshot { - existing_finalized_block: 0, - }) - .exit_code(), - EXIT_TERMINAL - ); - // Reader-level chain-id mismatch (warm-boot backstop) is terminal, like - // the boot-time BootstrapError::ChainIdMismatch. - assert_eq!( - RunError::Worker(WorkerExit::InputReader(InputReaderExit::Source( - InputReaderError::ChainIdMismatch { - rpc: 1, - expected: 31337, - } - ))) - .exit_code(), - EXIT_TERMINAL - ); - // The same mismatch surfacing during a startup-recovery safe-head sync - // (RecoveryError path) is terminal too — not the unclassified Recovery - // catch-all (which would loop on the wrong chain). - assert_eq!( - RunError::Bootstrap(BootstrapError::Recovery(RecoveryError::InputReader( - InputReaderError::ChainIdMismatch { - rpc: 1, - expected: 31337, - } - ))) - .exit_code(), - EXIT_TERMINAL - ); - } - - #[test] - fn r4_class_40_setup_needs_operator_recovery() { - // Sticky setup refusals: operator must run `setup --recovery`, not a - // plain restart (which would re-detect and re-refuse) — so they get a - // dedicated code, distinct from the auto-recovery class (10). - assert_eq!( - RunError::from(SetupRefuse::WalletNonceUnsettled { - pending: 14, - safe: 13, - }) - .exit_code(), - EXIT_SETUP_NEEDS_RECOVERY - ); - assert_eq!( - RunError::from(SetupRefuse::BatchPastCheckpoint { - checkpoint_block: 100, - found_block: 250, - safe_input_index: 7, - }) - .exit_code(), - EXIT_SETUP_NEEDS_RECOVERY - ); - // A checkpoint predating genesis is operator misconfig — terminal (30), - // not a recovery trigger. - assert_eq!( - RunError::Bootstrap(BootstrapError::CheckpointBeforeAppDeployment { - checkpoint_block: 5, - app_deployment_block: 10, - }) - .exit_code(), - EXIT_TERMINAL - ); - } - - #[test] - fn r4_class_1_unclassified() { - assert_eq!( - RunError::Io(std::io::Error::other("boom")).exit_code(), - EXIT_UNCLASSIFIED - ); - assert_eq!( - RunError::Worker(WorkerExit::Server(ServerExit::StoppedUnexpectedly)).exit_code(), - EXIT_UNCLASSIFIED - ); - assert_eq!( - RunError::from(FeeOracleError::Transient("RPC unavailable".into())).exit_code(), - EXIT_UNCLASSIFIED - ); - assert_eq!( - RunError::Worker(WorkerExit::FeeOracle(FeeOracleExit::Source( - FeeOracleError::Transient("RPC unavailable".into()), - ))) - .exit_code(), - EXIT_UNCLASSIFIED - ); - } - - #[test] - fn fee_oracle_fatal_math_is_terminal_on_bootstrap_and_worker() { - use crate::l1::fee_oracle::math::MathError; - assert_eq!( - RunError::from(FeeOracleError::FatalMath(MathError::Overflow)).exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::Worker(WorkerExit::FeeOracle(FeeOracleExit::Source( - FeeOracleError::FatalMath(MathError::Overflow), - ))) - .exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::Worker(WorkerExit::FeeOracle(FeeOracleExit::Source( - FeeOracleError::FatalMath(MathError::ExceedsRepresentableRange), - ))) - .exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::from(FeeOracleError::Misconfig("wrong pair".into())).exit_code(), - EXIT_TERMINAL - ); - assert_eq!( - RunError::Worker(WorkerExit::FeeOracle(FeeOracleExit::Source( - FeeOracleError::Misconfig("wrong pair".into()), - ))) - .exit_code(), - EXIT_TERMINAL - ); - } - - #[test] - fn fee_oracle_shutdown_ok_is_graceful() { - let result: Result, tokio::task::JoinError> = Ok(Ok(())); - assert!(FeeOracleExit::from_shutdown(result).is_ok()); - } -} diff --git a/sequencer/src/runtime/mod.rs b/sequencer/src/runtime/mod.rs index 8ba793f5..4b7a1cf6 100644 --- a/sequencer/src/runtime/mod.rs +++ b/sequencer/src/runtime/mod.rs @@ -1,507 +1,18 @@ // (c) Cartesi and individual authors (see AUTHORS) // SPDX-License-Identifier: Apache-2.0 (see LICENSE) -//! Process orchestration for the `run` subcommand, plus the shared -//! bootstrap helpers used by all three subcommands. +//! The runtime authority capabilities — the ADR's "structured process +//! ownership" mechanism, and nothing else: //! -//! The phase split: [`setup`] establishes the timeless deployment -//! state (identity, initial sync, genesis snapshot, `setup_complete` marker); -//! [`run`] boots workers from an already-set-up DB; [`flush`] settles the -//! wallet nonce. The CLI harness that dispatches them lives in -//! [`crate::harness`]. +//! - [`process_lock`] — the exclusive kernel-enforced data-directory lock +//! every command and nested blocking task retains until it truly stops. +//! - [`shutdown`] — `RuntimeScope` (lock + terminal-abort watchdog + +//! containment authority + fault recorder) and the slim cooperative +//! `ShutdownSignal`. //! -//! `run`'s phases: -//! -//! 1. **Gate + identity**: refuse unless `setup` completed; read the pinned -//! deployment identity from the DB (chain id / app address are no longer -//! CLI args — they come from the identity). -//! 2. **Preemptive recovery**: run the startup recovery procedure -//! ([`crate::recovery::run_preemptive_recovery`]). -//! 3. **Workers**: hand off to `workers::Workers` for spawn → select → -//! finish. -//! -//! Errors live in [`error`]; worker lifecycle in `workers`. +//! Everything here is consumed crate-wide (workers, egress, l1, recovery); +//! the command brackets live in [`crate::commands`], which also owns the +//! command-scoped config and error taxonomy. -pub mod clock; -pub mod config; -pub mod error; -pub mod flush; -pub mod setup; -mod setup_fill; +pub(crate) mod process_lock; pub mod shutdown; -#[cfg(test)] -pub(crate) mod test_support; -mod workers; - -use std::time::Duration; - -use crate::l1::reader::{InputReader, InputReaderConfig}; -use crate::storage::{self, DeploymentIdentity}; -use alloy_primitives::Address; -use config::{L1Config, RunConfig}; -use sequencer_core::application::Application; - -pub use error::{ - BatchSubmitterExit, BootstrapError, DangerDetectorExit, IdentityError, InputReaderExit, - LaneExit, RunError, ServerExit, SetupRecoveryError, SetupRefuse, WorkerExit, -}; - -use workers::{Workers, WorkersConfig}; - -pub(crate) const INPUT_READER_POLL_INTERVAL: Duration = Duration::from_secs(2); - -/// Boot the sequencer from an already-set-up DB. Generic over the app type -/// (for the lane's `from_dump`, the egress state-file path, and the -/// max-payload bound) but takes no app *value* — `setup` already registered -/// the genesis snapshot, so the lane reloads via `A::from_dump`. -pub async fn run(config: RunConfig) -> Result<(), RunError> -where - A: Application + Clone + Sync + 'static, -{ - // ── Gate + identity ────────────────────────────────────── - std::fs::create_dir_all(&config.data_dir)?; - let db_path = config.db_path(); - let timing = config.protocol_timing()?; - - // Refuse to boot unless `setup` completed; the identity it pinned - // supplies chain id / app address / InputBox address / app deployment block / - // submitter address — none of which are CLI args on `run`. - let identity = load_setup_identity(&db_path)?; - - // `run` holds the signing key (it submits). The key's address must match - // the pinned submitter address — running with the wrong key against a DB - // pinned to another submitter is a fail-loud identity mismatch. - let key = verify_submitter_key(config.resolve_private_key()?, &identity)?; - - // Validate the RPC chain id against the *pinned* chain id when L1 is - // reachable (guards against a wrong-chain RPC after setup, review F6); - // tolerate an unreachable L1 (warm boot — identity is already pinned). The - // tolerated case is backstopped by the input reader, which re-verifies the - // chain id on its first successful contact (`InputReaderConfig::expected_chain_id`), - // so a provider that reconnects on the wrong chain fails loud before - // ingesting any address-filtered foreign logs. - match validate_rpc_chain_id( - &config.eth_rpc_url, - identity.chain_id, - config.allow_insecure_rpc, - ) - .await - { - Ok(()) => {} - Err(RunError::Bootstrap(BootstrapError::ChainIdRpc { message })) => { - tracing::warn!( - error = %message, - "L1 unreachable at boot — continuing from pinned deployment identity" - ); - } - Err(other) => return Err(other), - } - - let l1_config = L1Config { - eth_rpc_url: config.eth_rpc_url.clone(), - input_box_address: identity.input_box_address, - app_address: identity.app_address, - batch_submitter_private_key: key, - batch_submitter_address: identity.batch_submitter_address, - chain_id: identity.chain_id, - allow_insecure_rpc: config.allow_insecure_rpc, - }; - - // Prefer a live refresh before recovery can reopen a Tip, but tolerate a - // transient L1/RPC failure the same way warm-boot tolerates unreachable - // chain-id checks: the price is already pinned by setup, and a single - // persisted max-age (`l1_read_stale_after`) bounds how long it may be - // retained at boot and in `run_forever`. Misconfig (wrong pool/pair/chain) - // stays terminal. Fixed mode needs no worker. - let max_price_age_ms = timing.l1_read_stale_after_secs().saturating_mul(1000); - let fee_oracle = match identity.fee_oracle { - storage::FeeOracleIdentity::Fixed { .. } => None, - storage::FeeOracleIdentity::Uniswap { - weth, - fee_token, - pool, - twap_window_secs, - } => { - let provider = crate::l1::provider::create_provider( - &config.eth_rpc_url, - config.allow_insecure_rpc, - ) - .map_err(|message| BootstrapError::FeeOracleMisconfig { message })?; - let uniswap = crate::l1::fee_oracle::UniswapConfig { - chain_id: identity.chain_id, - weth, - fee_token, - pool, - twap_window_secs, - }; - let poll = Duration::from_millis(config.fee_oracle.poll_interval_ms); - let oracle = match crate::l1::fee_oracle::UniswapV3PriceSource::connect( - provider.clone(), - uniswap, - ) - .await - { - Ok(token) => { - let oracle = crate::l1::fee_oracle::FeeOracle::new( - db_path.clone(), - poll, - max_price_age_ms, - provider, - Box::new(token), - ); - match oracle.refresh_once().await { - Ok(_) => oracle, - Err(crate::l1::fee_oracle::worker::FeeOracleError::Transient(message)) => { - tracing::warn!( - error = %message, - "fee oracle unreachable at boot — continuing from persisted price" - ); - crate::l1::fee_oracle::FeeOracle::ensure_persisted_price_fresh( - &db_path, - max_price_age_ms, - )?; - oracle - } - Err(error) => return Err(error.into()), - } - } - Err(error) => { - let (transient, message) = - crate::l1::fee_oracle::uniswap::bootstrap_price_source_error(error); - if !transient { - return Err(BootstrapError::FeeOracleMisconfig { message }.into()); - } - tracing::warn!( - error = %message, - "fee oracle unreachable at boot — continuing from persisted price" - ); - crate::l1::fee_oracle::FeeOracle::ensure_persisted_price_fresh( - &db_path, - max_price_age_ms, - )?; - crate::l1::fee_oracle::FeeOracle::reconnecting_uniswap( - db_path.clone(), - poll, - max_price_age_ms, - provider, - uniswap, - ) - } - }; - Some(oracle) - } - }; - - // `run` never re-discovers identity from L1 — it builds the reader from - // the pinned InputBox address + app deployment block and syncs incrementally. - let mut input_reader = InputReader::from_parts( - InputReaderConfig { - rpc_url: config.eth_rpc_url.clone(), - allow_insecure_rpc: config.allow_insecure_rpc, - app_address: identity.app_address, - poll_interval: INPUT_READER_POLL_INTERVAL, - long_block_range_error_codes: config.long_block_range_error_codes.clone(), - expected_chain_id: identity.chain_id, - }, - identity.input_box_address, - identity.app_deployment_block, - db_path.clone(), - identity.batch_submitter_address, - timing, - ); - - tracing::info!( - http_addr = %config.http_addr, - data_dir = %config.data_dir, - eth_rpc_url = %l1_config.eth_rpc_url, - input_box_address = %l1_config.input_box_address, - app_deployment_block = input_reader.app_deployment_block(), - chain_id = identity.chain_id, - app_address = %l1_config.app_address, - batch_submitter_address = %l1_config.batch_submitter_address, - max_wait_blocks = timing.max_wait_blocks, - preemptive_margin_blocks = timing.preemptive_margin_blocks, - danger_threshold = timing.danger_threshold(), - "sequencer startup" - ); - - // Always-load invariant, checked at the gate (before any recovery write): - // setup registers the genesis finalized snapshot, so a marker-present DB - // with no snapshot is a corrupt/incomplete setup. Fail loud here — ahead - // of preemptive recovery's DB mutations — rather than only at the lane. - { - let mut storage = storage::Storage::open(&db_path)?; - if storage.finalized_dump()?.is_none() { - return Err(BootstrapError::SetupNotComplete.into()); - } - } - - // ── Preemptive recovery ────────────────────────────────── - // See docs/recovery/ for the full design and TLA+ spec. - crate::recovery::run_preemptive_recovery(&db_path, &mut input_reader, &l1_config, &timing) - .await?; - - // ── Workers ────────────────────────────────────────────── - let domain = sequencer_core::build_input_domain(identity.chain_id, identity.app_address); - let mut workers = Workers::spawn::(WorkersConfig { - run_config: config, - l1_config, - timing, - input_reader, - domain, - fee_oracle, - }) - .await?; - - let first_exit = workers.select_first_exit().await; - workers.finish(first_exit).await -} - -// ── Bootstrap helpers (shared by setup / run / flush) ────────────────── - -pub(crate) fn batch_submitter_address_from_private_key( - private_key: &str, -) -> Result { - use alloy::signers::local::PrivateKeySigner; - use std::str::FromStr; - - Ok(PrivateKeySigner::from_str(private_key) - .map_err(|_| RunError::Io(std::io::Error::other("invalid private key")))? - .address()) -} - -/// Gate `run`/`flush` on a completed `setup` and return the pinned identity. -/// A missing marker — or a marker present but no identity (a corrupt / -/// incomplete setup) — is a terminal `SetupNotComplete`: the operator must -/// (re-)run `setup`, not retry `run`. -pub(crate) fn load_setup_identity(db_path: &str) -> Result { - let storage = storage::Storage::open(db_path)?; - if !storage.is_setup_complete()? { - return Err(BootstrapError::SetupNotComplete.into()); - } - match storage.deployment_identity()? { - Some(identity) => Ok(identity), - None => Err(BootstrapError::SetupNotComplete.into()), - } -} - -/// Verify that the RPC's `eth_chainId` matches the configured chain id. -/// -/// Treated as fatal on mismatch *and* on RPC error: pinning a wrong or -/// unverified chain id into storage would poison subsequent L1-unreachable -/// boots and issue soft confirmations against the wrong chain. Caller is -/// expected to retry on `ChainIdRpc`. -pub(crate) async fn validate_rpc_chain_id( - eth_rpc_url: &str, - expected: u64, - allow_insecure: bool, -) -> Result<(), RunError> { - use alloy::providers::Provider; - let check_provider = crate::l1::provider::create_provider(eth_rpc_url, allow_insecure) - .map_err(|e| RunError::Io(std::io::Error::other(e)))?; - match check_provider.get_chain_id().await { - Ok(rpc_chain_id) if rpc_chain_id != expected => { - Err(RunError::Bootstrap(BootstrapError::ChainIdMismatch { - rpc: rpc_chain_id, - config: expected, - })) - } - Ok(_) => Ok(()), - Err(e) => Err(RunError::Bootstrap(BootstrapError::ChainIdRpc { - message: e.to_string(), - })), - } -} - -pub(crate) fn ensure_deployment_identity( - db_path: &str, - expected: DeploymentIdentity, -) -> Result<(), RunError> { - let mut storage = storage::Storage::open(db_path)?; - if let Some(stored) = storage.deployment_identity()? { - return require_deployment_identity_match(stored, expected); - } - if storage.has_persisted_deployment_state()? { - return Err(IdentityError::OrphanedState.into()); - } - let stored = storage.load_or_insert_deployment_identity(expected)?; - require_deployment_identity_match(stored, expected) -} - -fn require_deployment_identity_match( - stored: DeploymentIdentity, - expected: DeploymentIdentity, -) -> Result<(), RunError> { - let fields = deployment_identity_mismatch_fields(stored, expected); - if fields.is_empty() { - return Ok(()); - } - Err(IdentityError::Mismatch { - fields: fields.join(", "), - stored: Box::new(stored), - expected: Box::new(expected), - } - .into()) -} - -/// Keyed-writer preflight shared by `run` and `flush`: confirm a resolved -/// batch-submitter signing `key` signs for the submitter `setup` pinned in -/// `identity`, returning the key on success. Both subcommands broadcast keyed -/// L1 txs, so signing under the wrong key would consume the wrong wallet's -/// nonce slots — a fail-loud identity mismatch, not a recoverable condition. -pub(crate) fn verify_submitter_key( - key: String, - identity: &DeploymentIdentity, -) -> Result { - let key_address = batch_submitter_address_from_private_key(&key)?; - if key_address != identity.batch_submitter_address { - let expected = DeploymentIdentity { - batch_submitter_address: key_address, - ..*identity - }; - require_deployment_identity_match(*identity, expected)?; - } - Ok(key) -} - -fn deployment_identity_mismatch_fields( - stored: DeploymentIdentity, - expected: DeploymentIdentity, -) -> Vec<&'static str> { - let mut fields = Vec::new(); - if stored.chain_id != expected.chain_id { - fields.push("chain_id"); - } - if stored.app_address != expected.app_address { - fields.push("app_address"); - } - if stored.input_box_address != expected.input_box_address { - fields.push("input_box_address"); - } - if stored.app_deployment_block != expected.app_deployment_block { - fields.push("app_deployment_block"); - } - if stored.batch_submitter_address != expected.batch_submitter_address { - fields.push("batch_submitter_address"); - } - if stored.fee_oracle != expected.fee_oracle { - fields.push("fee_oracle"); - } - fields -} - -#[cfg(test)] -mod tests { - use super::{ - BootstrapError, IdentityError, RunError, batch_submitter_address_from_private_key, - deployment_identity_mismatch_fields, ensure_deployment_identity, - require_deployment_identity_match, - }; - use crate::recovery::{RecoveryError, RefuseReason}; - use crate::storage::test_helpers::{SENDER_A, default_protocol_timing, temp_db}; - use crate::storage::{DeploymentIdentity, Storage}; - use alloy_primitives::Address; - use sequencer_core::protocol::ProtocolTimingError; - - // Margin/stale-boundary validation is exercised directly in - // `sequencer-core/src/protocol.rs`. The runtime tests below only cover - // the typed `From` conversions into `RunError` and the bootstrap-time - // identity guards. Worker `From` conversions live in - // `runtime/workers.rs`. - - #[test] - fn invalid_protocol_config_propagates_through_run_error() { - let err: RunError = ProtocolTimingError::MarginNotLessThanMaxWait { - margin: 1200, - max_wait: 1200, - } - .into(); - assert!(matches!( - err, - RunError::Bootstrap(BootstrapError::InvalidProtocolTiming(_)) - )); - } - - #[test] - fn startup_recovery_error_preserves_recovery_category() { - let err: RunError = RecoveryError::Refuse(RefuseReason::L1ViewStale).into(); - assert!(matches!( - err, - RunError::Bootstrap(BootstrapError::Recovery(RecoveryError::Refuse( - RefuseReason::L1ViewStale - ))) - )); - } - - fn identity() -> DeploymentIdentity { - DeploymentIdentity { - chain_id: 31337, - app_address: Address::repeat_byte(0x11), - input_box_address: Address::repeat_byte(0x22), - app_deployment_block: 42, - batch_submitter_address: Address::repeat_byte(0x33), - fee_oracle: crate::storage::FeeOracleIdentity::Fixed { log_gas_price: 0 }, - } - } - - #[test] - fn deployment_identity_match_accepts_same_identity() { - let identity = identity(); - require_deployment_identity_match(identity, identity).expect("same identity should match"); - } - - #[test] - fn deployment_identity_mismatch_reports_changed_fields() { - let stored = identity(); - let expected = DeploymentIdentity { - chain_id: 31338, - app_address: Address::repeat_byte(0x44), - batch_submitter_address: Address::repeat_byte(0x55), - ..stored - }; - - assert_eq!( - deployment_identity_mismatch_fields(stored, expected), - vec!["chain_id", "app_address", "batch_submitter_address"] - ); - let err = require_deployment_identity_match(stored, expected) - .expect_err("mismatch should refuse startup"); - assert!(matches!( - err, - RunError::Bootstrap(BootstrapError::Identity(IdentityError::Mismatch { fields, .. })) - if fields == "chain_id, app_address, batch_submitter_address" - )); - } - - #[test] - fn deployment_identity_refuses_non_empty_unpinned_db() { - let db = temp_db("runtime-unpinned-deployment-state"); - { - let mut storage = Storage::open(db.path.as_str()).expect("open storage"); - storage - .append_safe_inputs(0, &[], SENDER_A, &default_protocol_timing()) - .expect("seed deployment-bound state"); - } - - let err = ensure_deployment_identity(db.path.as_str(), identity()) - .expect_err("non-empty unpinned DB must refuse"); - assert!(matches!( - err, - RunError::Bootstrap(BootstrapError::Identity(IdentityError::OrphanedState)) - )); - } - - #[test] - fn invalid_private_key_error_does_not_echo_key_material() { - let secret = "0xabc123SECRET"; - let err = batch_submitter_address_from_private_key(secret) - .expect_err("invalid private key should be rejected"); - let message = err.to_string(); - - assert_eq!(message, "invalid private key"); - assert!( - !message.contains(secret), - "private key material must not be reflected in startup errors" - ); - } -} diff --git a/sequencer/src/runtime/process_lock.rs b/sequencer/src/runtime/process_lock.rs new file mode 100644 index 00000000..1ebd52f8 --- /dev/null +++ b/sequencer/src/runtime/process_lock.rs @@ -0,0 +1,243 @@ +// (c) Cartesi and individual authors (see AUTHORS) +// SPDX-License-Identifier: Apache-2.0 (see LICENSE) + +//! Exclusive data-directory process lock. +//! +//! Every subcommand that touches a data directory (`run`, `setup`, +//! `flush-mempool`) acquires an OS-held advisory +//! lock on `/sequencer.lock` before reading or mutating anything +//! there. `run` transfers it into the structured runtime scope, which retains +//! it until every runtime-owned data-dir task has stopped; nested blocking +//! work retains a clone, including during pre-worker setup/recovery. The other +//! commands hold it until they and any detached blocking work return. +//! Persisted lifecycle rows cannot distinguish a live owner from a stale one; +//! a kernel-held lock can — it vanishes with the process, however the process +//! dies. This prevents two processes from racing settlement, rebuild, or +//! boot on one data dir, and is the exclusive-ownership primitive the +//! authority-boundary ADR's durable lifecycle builds on. A non-owning weak +//! witness to the same descriptor also drives terminal shutdown: after two +//! seconds, a live witness means some runtime-owned work has not drained and +//! the process aborts. + +use std::fs::{File, TryLockError}; +use std::path::Path; +use std::sync::{Arc, Weak}; + +use thiserror::Error; + +/// The lock module's own typed error: `runtime/` is the capability +/// substrate and must not import the command layer's error taxonomy. +/// `commands::error` converts this into its `BootstrapError` classes. +#[derive(Debug, Error)] +pub(crate) enum ProcessLockError { + /// Another live process holds the exclusive data-directory lock. + /// Retry-safe: an orchestrated restart racing the previous owner's + /// drain resolves on its own. + #[error( + "another process holds the data-directory lock ({path}); \ + refusing to run concurrently" + )] + Locked { path: String }, + #[error(transparent)] + Io(#[from] std::io::Error), +} + +/// Lock-anchor file name inside the data directory. Contents are irrelevant +/// and the file is never deleted: unlinking a lock file invites the classic +/// unlink/re-open race where two processes hold locks on different inodes of +/// the same path. +const LOCK_FILE_NAME: &str = "sequencer.lock"; + +/// A held exclusive lock on a data directory. Clones retain the same locked +/// descriptor; the lock is released only after the last clone drops (or on +/// process death). This lets detached blocking work retain exclusivity even +/// if the async command future awaiting it is cancelled. +#[derive(Clone, Debug)] +pub(crate) struct ProcessLock { + /// Keeps the locked descriptor open; the OS lock lives on it. + _file: Arc, +} + +/// Non-owning observation of a command/runtime lifetime. The terminal abort +/// watchdog uses this to distinguish a completed drain from work that still +/// owns the data directory without retaining the lock itself. +#[derive(Clone, Debug)] +pub(crate) struct ProcessLockWitness { + file: Weak, +} + +impl ProcessLockWitness { + pub(crate) fn is_held(&self) -> bool { + self.file.upgrade().is_some() + } +} + +impl ProcessLock { + /// Acquire the exclusive data-directory lock without blocking. + /// [`ProcessLockError::Locked`] means another live process owns the + /// directory — refusing here is what makes "one process per data dir" a + /// kernel-enforced fact rather than an operational convention. + pub(crate) fn acquire(data_dir: &str) -> Result { + let path = Path::new(data_dir).join(LOCK_FILE_NAME); + let file = File::options() + .read(true) + .write(true) + .create(true) + .truncate(false) + .open(&path)?; + match file.try_lock() { + Ok(()) => Ok(Self { + _file: Arc::new(file), + }), + Err(TryLockError::WouldBlock) => Err(ProcessLockError::Locked { + path: path.display().to_string(), + }), + Err(TryLockError::Error(err)) => Err(err.into()), + } + } + + /// Test lock on a leaked temp dir (bounded by test count): component + /// tests need a held lock now that data-dir workers require one at + /// construction. + #[cfg(test)] + pub(crate) fn test() -> Self { + let dir = tempfile::tempdir().expect("test lock tempdir"); + let lock = + Self::acquire(dir.path().to_str().expect("utf8 path")).expect("test lock acquire"); + std::mem::forget(dir); + lock + } + + /// Return a non-owning witness for the lifetime of this lock and all of + /// its clones. Observing the witness never extends that lifetime. + pub(crate) fn witness(&self) -> ProcessLockWitness { + ProcessLockWitness { + file: Arc::downgrade(&self._file), + } + } +} + +/// Spawn blocking work while retaining the command/runtime lock for the +/// closure's real lifetime. Dropping the async join handle detaches Tokio +/// blocking work; this wrapper prevents that detach from releasing exclusive +/// data-directory ownership prematurely. The lock is required: a data-dir +/// blocking task without ownership is unrepresentable. +pub(crate) fn spawn_blocking_with_lock( + process_lock: ProcessLock, + work: F, +) -> tokio::task::JoinHandle +where + F: FnOnce() -> R + Send + 'static, + R: Send + 'static, +{ + tokio::task::spawn_blocking(move || { + let _process_lock = process_lock; + work() + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn second_acquire_refuses_while_held_and_succeeds_after_release() { + let dir = tempfile::tempdir().expect("tempdir"); + let data_dir = dir.path().to_str().expect("utf8 path"); + + let held = ProcessLock::acquire(data_dir).expect("first acquire"); + let refused = + ProcessLock::acquire(data_dir).expect_err("a held lock must refuse a second owner"); + assert!( + matches!(&refused, ProcessLockError::Locked { .. }), + "expected Locked, got {refused:?}" + ); + // The command layer classifies contention retry-safe: the previous + // owner may be draining. + let projected = crate::commands::error::CommandError::from(refused); + assert_eq!( + projected.exit_code(), + crate::commands::error::EXIT_RESTART_TRANSIENT, + ); + + drop(held); + ProcessLock::acquire(data_dir).expect("acquire after release"); + } + + #[test] + fn locks_on_distinct_data_dirs_are_independent() { + let dir_a = tempfile::tempdir().expect("tempdir a"); + let dir_b = tempfile::tempdir().expect("tempdir b"); + let _a = ProcessLock::acquire(dir_a.path().to_str().expect("utf8")).expect("lock a"); + let _b = ProcessLock::acquire(dir_b.path().to_str().expect("utf8")).expect("lock b"); + } + + #[test] + fn clone_retains_exclusivity_after_original_drops() { + let dir = tempfile::tempdir().expect("tempdir"); + let data_dir = dir.path().to_str().expect("utf8 path"); + let original = ProcessLock::acquire(data_dir).expect("acquire"); + let retained = original.clone(); + + drop(original); + ProcessLock::acquire(data_dir).expect_err("retained clone must keep the lock held"); + + drop(retained); + ProcessLock::acquire(data_dir).expect("last clone releases the lock"); + } + + #[test] + fn weak_witness_tracks_the_last_lock_owner_without_retaining_it() { + let dir = tempfile::tempdir().expect("tempdir"); + let data_dir = dir.path().to_str().expect("utf8 path"); + let original = ProcessLock::acquire(data_dir).expect("acquire"); + let retained = original.clone(); + let witness = original.witness(); + + assert!(witness.is_held()); + drop(original); + assert!( + witness.is_held(), + "a retained clone still owns the lifetime" + ); + drop(retained); + assert!( + !witness.is_held(), + "the weak witness must not retain the lock" + ); + } + + #[tokio::test] + async fn detached_blocking_work_retains_exclusivity_until_it_stops() { + let dir = tempfile::tempdir().expect("tempdir"); + let data_dir = dir.path().to_str().expect("utf8 path").to_string(); + let held = ProcessLock::acquire(&data_dir).expect("acquire"); + let (started_tx, started_rx) = tokio::sync::oneshot::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + + let task = spawn_blocking_with_lock(held.clone(), move || { + let _ = started_tx.send(()); + release_rx.recv().expect("release blocking work"); + }); + started_rx.await.expect("blocking work started"); + drop(task); // detach the running blocking closure + drop(held); + + ProcessLock::acquire(&data_dir) + .expect_err("detached blocking work must retain process ownership"); + release_tx.send(()).expect("release blocking work"); + + // The closure has no join handle now, so poll the kernel lock with a + // bounded yield loop rather than racing its final drop. + tokio::time::timeout(std::time::Duration::from_secs(1), async { + loop { + if ProcessLock::acquire(&data_dir).is_ok() { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("blocking closure should release its retained lock"); + } +} diff --git a/sequencer/src/runtime/shutdown.rs b/sequencer/src/runtime/shutdown.rs index 22e82153..06202899 100644 --- a/sequencer/src/runtime/shutdown.rs +++ b/sequencer/src/runtime/shutdown.rs @@ -1,17 +1,271 @@ // (c) Cartesi and individual authors (see AUTHORS) // SPDX-License-Identifier: Apache-2.0 (see LICENSE) +//! Shutdown signalling + terminal-fault containment inside a pre-armed run. +//! +//! Containment is classification-at-birth, in this order: +//! +//! - [`RuntimeScope::contain_storage_invariant_failure`]: elect the first +//! reporter (CAS), set the sticky containment bit, arm the terminal abort +//! watchdog, request shutdown, then invoke the durable recorder (the +//! black box's terminal-cause row, installed at worker spawn). The watchdog +//! precedes both cancellation and recording because either may block. +//! - [`ShutdownSignal::is_storage_invariant_contained`]: checked by +//! externalization sites (acks, L1 sends, WS frames, snapshot stream +//! starts) before emitting; set only by containment, so a missed check is +//! bounded by the exit contract (terminal exits are not restarted) and +//! by the I15 freeze triggers on the tables they cover — partial +//! backstops, not a barrier. +//! +//! Honest bounds: the black box's terminal-cause row is best-effort +//! telemetry (restart policy is the exit contract, and a persistent fault +//! re-detects fail-loud on the next boot that reads it — there is no +//! database boot gate to keep durable). +//! In a process-lock-backed runtime, terminal containment gives the complete +//! runtime lifetime two seconds to drain before aborting the process. Ordinary +//! operator/recovery shutdown remains cooperatively unbounded. + use std::sync::Arc; use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, Instant}; use tokio::sync::Notify; +use super::process_lock::{ProcessLock, ProcessLockWitness}; + +const TERMINAL_ABORT_TIMEOUT: Duration = Duration::from_secs(2); + +/// Durable terminal-fault recorder installed at worker spawn (appends the +/// black box's terminal-cause row). +pub(crate) type FaultRecorder = Arc; + +type AbortAction = Arc; + +#[derive(Clone)] +struct TerminalAbortWatchdog { + runtime_lifetime: ProcessLockWitness, + timeout: Duration, + abort_action: AbortAction, +} + +impl TerminalAbortWatchdog { + fn production(runtime_lifetime: ProcessLockWitness) -> Self { + Self { + runtime_lifetime, + timeout: TERMINAL_ABORT_TIMEOUT, + abort_action: Arc::new(|| std::process::abort()), + } + } + + fn arm(&self) { + let runtime_lifetime = self.runtime_lifetime.clone(); + let deadline = Instant::now() + self.timeout; + let abort_action = self.abort_action.clone(); + let spawn_failure_abort = self.abort_action.clone(); + if let Err(error) = std::thread::Builder::new() + .name("sequencer-terminal-abort".into()) + .spawn(move || { + std::thread::sleep(deadline.saturating_duration_since(Instant::now())); + if runtime_lifetime.is_held() { + abort_action(); + } + }) + { + // Losing the independent timer would silently discard the hard + // terminal-shutdown bound. Abort before attempting synchronous + // logging: a blocked subscriber must not defeat fail-closed + // behavior under the same resource pressure that prevented the + // watchdog thread from starting. Production's action never + // returns; test actions may, so retain the diagnostic afterward. + spawn_failure_abort(); + tracing::error!(%error, "failed to arm terminal abort watchdog"); + } + } +} + +/// Cooperative shutdown notification — exactly what the name says, and +/// nothing else. Freely `Default`-constructible; carries no authority. #[derive(Clone, Default)] pub struct ShutdownSignal { is_shutting_down: Arc, notify: Arc, } +/// The command/runtime-lifetime capability the ADR calls `RuntimeScope`: +/// exclusive data-directory ownership plus terminal-fault containment. Only +/// constructible from a held [`ProcessLock`] — installed at lock +/// acquisition, before any signal or worker exists — so a watchdog-less or +/// lock-less containment object is unrepresentable in production. +/// Workers that touch the data directory or externalize take a scope; +/// pure-notification consumers take its [`ShutdownSignal`]. +#[derive(Clone)] +pub struct RuntimeScope { + signal: ShutdownSignal, + /// Single containment authority: winning this `OnceLock` *is* the sticky + /// containment bit, so the bit and its cause become visible together — + /// there is no window where containment reads true with no cause. + first_containment_cause: Arc>, + /// Durable fault recorder (the black box's terminal-cause row), installed + /// once during runtime preparation. Invoked only after the containment + /// bit, watchdog, and shutdown request — recording can block, and no new + /// externalization may be authorized while it runs. Best-effort + /// telemetry: when it fails, the cause is still in the logs, the process + /// still exits terminal, and a persistent fault re-detects on the next + /// boot that reads it. + fault_recorder: Arc>, + terminal_abort_watchdog: TerminalAbortWatchdog, + /// Runtime-lifetime ownership. Every scope clone retains the lock; + /// nested blocking tasks retain their own clone through + /// [`crate::runtime::process_lock::spawn_blocking_with_lock`], so + /// data-directory exclusivity outlives detached work. + process_lock: ProcessLock, +} + +impl RuntimeScope { + /// Create the scope that owns a command/runtime lifetime. The process + /// lock is shared by every clone and released only after the final owner + /// drops it, including on partial startup failure or caller cancellation. + pub(crate) fn new(process_lock: ProcessLock) -> Self { + Self { + signal: ShutdownSignal::default(), + first_containment_cause: Arc::default(), + fault_recorder: Arc::default(), + terminal_abort_watchdog: TerminalAbortWatchdog::production(process_lock.witness()), + process_lock, + } + } + + #[cfg(test)] + fn with_test_terminal_abort_watchdog( + process_lock: ProcessLock, + timeout: Duration, + abort_action: AbortAction, + ) -> Self { + Self { + signal: ShutdownSignal::default(), + first_containment_cause: Arc::default(), + fault_recorder: Arc::default(), + terminal_abort_watchdog: TerminalAbortWatchdog { + runtime_lifetime: process_lock.witness(), + timeout, + abort_action, + }, + process_lock, + } + } + + /// The pure notification half, for consumers that only wait for stop. + pub(crate) fn signal(&self) -> ShutdownSignal { + self.signal.clone() + } + + /// The held data-directory lock, for nested blocking work. + pub(crate) fn process_lock(&self) -> ProcessLock { + self.process_lock.clone() + } + + pub fn request_shutdown(&self) { + self.signal.request_shutdown(); + } + + pub fn is_shutdown_requested(&self) -> bool { + self.signal.is_shutdown_requested() + } + + pub async fn wait_for_shutdown(&self) { + self.signal.wait_for_shutdown().await; + } + + /// Contain a persistent storage invariant failure: CAS-elect the first + /// reporter, set the sticky containment bit, arm the terminal watchdog, + /// request shutdown, then invoke the durable recorder (the black box's + /// terminal-cause row, best-effort). Sync — callable from any thread, + /// async or blocking. + /// + /// This is containment, not recovery: the supervisor maps the contained + /// state to the terminal exit class (30 — do not restart, page). A + /// persistent fault re-detects fail-loud on any boot that reads it; the + /// black-box row is the cause's telemetry, not a boot gate. + pub(crate) fn contain_storage_invariant_failure(&self, cause: impl Into) { + // First-winner election: setting the cause is the containment bit, + // so exactly one reporter proceeds and echoes return immediately + // (their causes are already in the error logs). + if self.first_containment_cause.set(cause.into()).is_err() { + return; + } + // The independent watchdog must precede both cooperative cancellation + // and audit recording: either may block while runtime work retains the + // process-lifetime capability. + self.terminal_abort_watchdog.arm(); + self.request_shutdown(); + if let Some(recorder) = self.fault_recorder.get() { + recorder( + self.first_containment_cause + .get() + .expect("cause was just installed by the elected reporter"), + ); + } + } + + /// Install the durable recorder (once; later installs are ignored). + pub(crate) fn set_fault_recorder(&self, recorder: FaultRecorder) { + let _ = self.fault_recorder.set(recorder); + } + + /// Whether a terminal fault has been contained. Externalization sites + /// (acks, sends, frames, streams) check this before emitting — through + /// [`Self::authorize`], whose token their effect functions require. True + /// iff [`Self::containment_cause`] is present — one authority, no window. + pub(crate) fn is_storage_invariant_contained(&self) -> bool { + self.first_containment_cause.get().is_some() + } + + pub(crate) fn containment_cause(&self) -> Option<&str> { + self.first_containment_cause.get().map(String::as_str) + } + + /// Consult containment and mint the externalization token: `None` once a + /// terminal fault is contained. The authority-bearing effect functions + /// (acknowledge, L1 send, WS emit, snapshot-stream start) take an + /// [`Authorized`], so a new externalization site cannot forget the check + /// — the compile error replaces the convention. This is the honest + /// bounded-lag consult, not a fence: a token minted before a concurrent + /// containment may finish its already-authorized effect (ADR). + pub(crate) fn authorize(&self) -> Option> { + if self.is_storage_invariant_contained() { + None + } else { + Some(Authorized { + _scope: std::marker::PhantomData, + }) + } + } +} + +/// Zero-sized, borrow-scoped proof that terminal containment was consulted +/// and found clear at this effect boundary. Obtainable only from +/// [`RuntimeScope::authorize`]; carries no runtime state — this is the +/// type-level obligation the ADR's rejected `EffectGate` was not (no mutex, +/// no actor, no second state machine). +#[derive(Clone, Copy)] +pub(crate) struct Authorized<'scope> { + _scope: std::marker::PhantomData<&'scope RuntimeScope>, +} + +/// Test scope: a leaked temp-dir lock (bounded by test count) plus a no-op +/// abort watchdog, so containing a fault in a component test neither needs a +/// data directory nor risks aborting the test binary at the 2s deadline. +#[cfg(test)] +impl Default for RuntimeScope { + fn default() -> Self { + let dir = tempfile::tempdir().expect("test scope tempdir"); + let lock = + ProcessLock::acquire(dir.path().to_str().expect("utf8 path")).expect("test scope lock"); + std::mem::forget(dir); + Self::with_test_terminal_abort_watchdog(lock, TERMINAL_ABORT_TIMEOUT, Arc::new(|| {})) + } +} + impl ShutdownSignal { pub fn request_shutdown(&self) { let was_shutting_down = self.is_shutting_down.swap(true, Ordering::SeqCst); @@ -41,3 +295,156 @@ impl ShutdownSignal { } } } + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::Mutex; + + const TEST_WATCHDOG_TIMEOUT: Duration = Duration::from_millis(25); + + fn signal_with_test_watchdog(abort_action: AbortAction) -> RuntimeScope { + let dir = tempfile::tempdir().expect("tempdir"); + let lock = ProcessLock::acquire(dir.path().to_str().expect("utf8 path")).expect("lock"); + // The held descriptor remains valid after the temporary directory is + // removed; the path itself is irrelevant to the lifetime witness. + RuntimeScope::with_test_terminal_abort_watchdog(lock, TEST_WATCHDOG_TIMEOUT, abort_action) + } + + #[test] + fn containment_elects_first_reporter_and_closes_before_recording() { + let signal = RuntimeScope::default(); + let recorded: Arc>> = Arc::default(); + signal.set_fault_recorder({ + let signal_view = signal.clone(); + let recorded = recorded.clone(); + Arc::new(move |cause| { + // The bit must already be visible while the (possibly slow) + // durable recording runs: no new externalization is + // authorized during the lifecycle recorder write. + assert!(signal_view.is_storage_invariant_contained()); + assert!(signal_view.is_shutdown_requested()); + recorded.lock().unwrap().push(cause.to_string()); + }) + }); + + assert!(!signal.is_storage_invariant_contained()); + signal.contain_storage_invariant_failure("first cause"); + signal.contain_storage_invariant_failure("second cause (echo)"); + + assert!(signal.is_storage_invariant_contained()); + // The CAS elects exactly one reporter; echoes return immediately. + assert_eq!(*recorded.lock().unwrap(), vec!["first cause".to_string()]); + } + + #[test] + fn plain_shutdown_is_not_containment() { + let signal = RuntimeScope::default(); + signal.request_shutdown(); + assert!(signal.is_shutdown_requested()); + assert!(!signal.is_storage_invariant_contained()); + } + + #[test] + fn watchdog_fires_while_the_fault_recorder_is_blocked() { + let (abort_tx, abort_rx) = std::sync::mpsc::channel(); + let signal = signal_with_test_watchdog(Arc::new(move || { + let _ = abort_tx.send(()); + })); + let recorder_entered = Arc::new(std::sync::Barrier::new(2)); + let release_recorder = Arc::new(std::sync::Barrier::new(2)); + let recorder_entered_for_callback = recorder_entered.clone(); + let release_recorder_for_callback = release_recorder.clone(); + signal.set_fault_recorder(Arc::new(move |_| { + recorder_entered_for_callback.wait(); + release_recorder_for_callback.wait(); + })); + + let reporter = { + let signal = signal.clone(); + std::thread::spawn(move || signal.contain_storage_invariant_failure("terminal fault")) + }; + recorder_entered.wait(); + abort_rx + .recv_timeout(Duration::from_secs(1)) + .expect("watchdog must fire while recorder remains blocked"); + assert!(signal.is_storage_invariant_contained()); + assert!(signal.is_shutdown_requested()); + + release_recorder.wait(); + reporter.join().expect("reporter thread"); + } + + #[test] + fn watchdog_noops_after_every_runtime_owner_drops() { + let (abort_tx, abort_rx) = std::sync::mpsc::channel(); + let signal = signal_with_test_watchdog(Arc::new(move || { + let _ = abort_tx.send(()); + })); + + signal.contain_storage_invariant_failure("terminal fault"); + drop(signal); + + assert!( + abort_rx.recv_timeout(Duration::from_millis(250)).is_err(), + "a completed runtime drain must suppress the abort" + ); + } + + #[test] + fn retained_lock_clone_keeps_the_watchdog_live_after_signal_drops() { + let dir = tempfile::tempdir().expect("tempdir"); + let lock = ProcessLock::acquire(dir.path().to_str().expect("utf8 path")).expect("lock"); + let nested_work_lock = lock.clone(); + let (abort_tx, abort_rx) = std::sync::mpsc::channel(); + let signal = RuntimeScope::with_test_terminal_abort_watchdog( + lock, + TEST_WATCHDOG_TIMEOUT, + Arc::new(move || { + let _ = abort_tx.send(()); + }), + ); + + signal.contain_storage_invariant_failure("terminal fault"); + drop(signal); + + abort_rx + .recv_timeout(Duration::from_secs(1)) + .expect("nested work retaining the process lock must trigger abort"); + drop(nested_work_lock); + } + + #[test] + fn first_reporter_arms_exactly_one_watchdog() { + let (abort_tx, abort_rx) = std::sync::mpsc::channel(); + let signal = signal_with_test_watchdog(Arc::new(move || { + let _ = abort_tx.send(()); + })); + + signal.contain_storage_invariant_failure("first cause"); + signal.contain_storage_invariant_failure("echo"); + abort_rx + .recv_timeout(Duration::from_secs(1)) + .expect("first watchdog action"); + assert!( + abort_rx.recv_timeout(Duration::from_millis(100)).is_err(), + "an echo must not arm another deadline" + ); + assert_eq!(signal.containment_cause(), Some("first cause")); + } + + #[test] + fn ordinary_shutdown_does_not_arm_the_terminal_watchdog() { + let (abort_tx, abort_rx) = std::sync::mpsc::channel(); + let signal = signal_with_test_watchdog(Arc::new(move || { + let _ = abort_tx.send(()); + })); + + signal.request_shutdown(); + + assert!( + abort_rx.recv_timeout(Duration::from_millis(250)).is_err(), + "ordinary operator shutdown remains unbounded and graceful" + ); + } +} diff --git a/sequencer/src/runtime/workers.rs b/sequencer/src/runtime/workers.rs deleted file mode 100644 index 4e948e08..00000000 --- a/sequencer/src/runtime/workers.rs +++ /dev/null @@ -1,766 +0,0 @@ -// (c) Cartesi and individual authors (see AUTHORS) -// SPDX-License-Identifier: Apache-2.0 (see LICENSE) - -//! Runtime worker lifecycle: spawn → run-until-first-exit → orderly cleanup. -//! -//! [`Workers`] owns the core runtime worker handles plus an optional live -//! Uniswap fee-oracle worker; fixed pricing has no worker. -//! Three methods describe its lifecycle: -//! -//! - [`Workers::spawn`]: build all configs, spawn workers, return owning struct. -//! - [`Workers::select_first_exit`]: race the workers + OS shutdown signal, -//! return whichever fired first. -//! - [`Workers::finish`]: request shutdown, await each component (logging -//! cleanup-time errors), surface the primary failure. -//! -//! Worker plumbing is intentionally explicit per-worker (6 fields, 6 spawn -//! statements, 6 select arms, 6 cleanup entries). Adding a seventh worker means -//! editing each of those four sites — but each edit is obvious and local. - -use std::future::Future; -use std::pin::Pin; -use std::sync::Arc; -use std::time::Duration; - -use alloy::providers::DynProvider; -use tokio::task::JoinHandle; -use tracing::warn; - -use crate::egress::l2_tx_feed::{L2TxFeed, L2TxFeedConfig}; -use crate::http::{self, ApiConfig}; -use crate::ingress::inclusion_lane::{ - InclusionLane, InclusionLaneConfig, InclusionLaneError, dump_info, dump_info::delete_dump_dir, -}; -use crate::l1::fee_oracle::FeeOracle; -use crate::l1::reader::{InputReader, InputReaderError}; -use crate::l1::submitter::{ - BatchPosterConfig, BatchSubmitter, BatchSubmitterConfig, BatchSubmitterError, - EthereumBatchPoster, SubmitterExit, -}; -use crate::recovery::{DangerDetector, DangerDetectorError, DetectorExit}; -use crate::runtime::config::{L1Config, RunConfig}; -use crate::runtime::error::{ - BatchSubmitterExit, DangerDetectorExit, FeeOracleExit, InputReaderExit, LaneExit, RunError, - ServerExit, WorkerExit, -}; -use crate::runtime::shutdown::ShutdownSignal; -use sequencer_core::application::Application; -use sequencer_core::protocol::ProtocolTiming; - -const QUEUE_CAPACITY: usize = 8192; -/// Danger detector cadence. Cheap DB-only check; re-running quickly bounds the -/// lag on entering the danger zone. The preemptive margin absorbs bounded lag. -const DANGER_DETECTOR_POLL_INTERVAL: Duration = Duration::from_secs(2); - -/// Which event ended the `select!` race in [`Workers::select_first_exit`]. -pub(crate) enum FirstExit { - Signal(Option), - Worker(WorkerExit), -} - -/// Inputs to [`Workers::spawn`]. Consumed entirely; the caller has nothing -/// further to do with these after the call. -/// -/// No genesis app instance: `setup` already registered the finalized genesis -/// snapshot, so the lane reloads via `A::from_dump`. The `domain` is built by -/// `run` from the pinned deployment identity. -pub(crate) struct WorkersConfig { - pub run_config: RunConfig, - pub l1_config: L1Config, - pub timing: ProtocolTiming, - pub input_reader: InputReader, - pub domain: alloy_sol_types::Eip712Domain, - pub fee_oracle: Option, -} - -/// Owns the runtime worker handles + the shutdown signal that drives them. -/// Construction (`spawn`) and teardown (`finish`) bracket the worker -/// lifecycle. -pub(crate) struct Workers { - server: JoinHandle>, - lane: JoinHandle>, - reader: JoinHandle>, - submitter: JoinHandle>, - detector: JoinHandle>, - fee_oracle: Option>>, - shutdown: ShutdownSignal, -} - -impl Workers { - /// Build the worker configs, spawn each worker, return the owning struct. - /// Logs `listening` once the HTTP server is bound. - pub(crate) async fn spawn( - cfg: WorkersConfig, - ) -> Result { - let WorkersConfig { - run_config, - l1_config, - timing, - input_reader, - domain, - fee_oracle, - } = cfg; - - // Derived values — kept inside `spawn` so `WorkersConfig` stays - // minimal and these aren't computed twice in the caller. - let db_path = run_config.db_path(); - let app_deployment_block = input_reader.app_deployment_block(); - - let shutdown = ShutdownSignal::default(); - - // Inclusion lane: takes the app, returns the tx-sender the HTTP - // ingress route will publish to. - let mut storage = crate::storage::Storage::open(&db_path)?; - let dumps_dir = std::path::Path::new(&run_config.data_dir).join("dumps"); - std::fs::create_dir_all(&dumps_dir)?; - - // Structural startup, in this order: - // - // 1. Reset stale leases. A crashed previous run may have left - // `lease_count > 0` on dumps that aren't being read by - // anyone now; without this, GC would skip them forever. - // 2. Require the finalized snapshot (always-load invariant). `setup` - // registered the genesis snapshot and `run` gated on the - // `setup_complete` marker, so it must be present — a missing one - // is a terminal incomplete-setup, not a cold-start to paper over - // (run holds no genesis app instance). - // 3. Ensure an open Tip exists (tip-existence invariant). The policy - // price was already written by setup (Fixed) or by run before - // preemptive recovery (Uniswap), so this samples a valid fee. - // Opens the - // genesis Tip on a fresh DB, no-op otherwise. The lane loads the - // head itself after catch-up; this step only establishes the - // invariant. Runs after preemptive recovery has synced the safe - // head, so the genesis frame's `safe_block` dates to startup (full - // landing budget), not to an earlier, possibly stale view. (This - // is a B-time quantity — it stays in `run`, never in `setup`.) - // 4. GC SQLite-side: drop any rows now unreferenced after - // promotions or invalidations that finalized just before - // the previous shutdown. - // 5. Orphan FS sweep: remove directories under `dumps_dir` - // that aren't tracked by SQLite (crash-during-create_dump - // or crash-during-GC-after-row-delete artifacts). - storage.reset_dump_leases()?; - require_finalized_snapshot(&mut storage)?; - restamp_finalized_promotion(&mut storage)?; - storage.ensure_open_tip()?; - let gc_removed = snapshot_gc_at_startup::(&mut storage)?; - let sweep_removed = sweep_orphan_dumps::(&mut storage, &dumps_dir)?; - tracing::debug!( - gc_removed, - sweep_removed, - "snapshot startup cleanup complete", - ); - - let (tx, lane) = InclusionLane::::start( - QUEUE_CAPACITY, - shutdown.clone(), - storage, - InclusionLaneConfig::new(l1_config.batch_submitter_address, dumps_dir) - .with_max_batch_open(run_config.max_batch_open()), - ); - - // Input reader: produces safe-input rows from L1. - let reader = input_reader.start(shutdown.clone())?; - - // Batch submitter: posts closed batches to L1. - let poster_config = BatchPosterConfig { - l1_submit_address: l1_config.input_box_address, - app_address: l1_config.app_address, - batch_submitter_address: l1_config.batch_submitter_address, - start_block: app_deployment_block, - confirmation_depth: run_config.batch_submitter_confirmation_depth, - seconds_per_block: run_config.timing.seconds_per_block, - long_block_range_error_codes: run_config.long_block_range_error_codes.clone(), - expected_chain_id: l1_config.chain_id, - }; - let provider = build_batch_submitter_provider(&l1_config)?; - let poster = Arc::new(EthereumBatchPoster::new(provider, poster_config)); - let submitter_config = BatchSubmitterConfig { - idle_poll_interval_ms: run_config.batch_submitter_idle_poll_interval_ms, - }; - let submitter = BatchSubmitter::new(db_path.clone(), poster, submitter_config) - .start(shutdown.clone())?; - - // Danger detector: trips startup recovery on bad DB/L1 state. - let detector = DangerDetector::new(db_path.clone(), timing, DANGER_DETECTOR_POLL_INTERVAL) - .start(shutdown.clone())?; - - let fee_oracle = fee_oracle.map(|oracle| oracle.start(shutdown.clone())); - - // HTTP server (ingress /tx + egress /ws/subscribe + /health, currently merged). - let tx_feed = L2TxFeed::new( - db_path.clone(), - shutdown.clone(), - L2TxFeedConfig { - batch_submitter_address: Some(l1_config.batch_submitter_address), - ..L2TxFeedConfig::default() - }, - ); - let server = http::start( - &run_config.http_addr, - tx, - domain, - A::MAX_METHOD_PAYLOAD_BYTES, - shutdown.clone(), - tx_feed, - ApiConfig::default(), - http::SnapshotState { - db_path: db_path.clone(), - // The DB row stores the dump *directory*; the app's state - // file lives under its `state` subtree. - state_file_in_dump: |dump_dir| { - A::state_file_in_dump(&crate::ingress::inclusion_lane::dump_info::app_prefix( - dump_dir, - )) - }, - }, - ) - .await?; - tracing::info!(address = %run_config.http_addr, "listening"); - - Ok(Self { - server, - lane, - reader, - submitter, - detector, - fee_oracle, - shutdown, - }) - } - - /// Race ctrl_c against each worker's join handle. The first to complete - /// produces the [`FirstExit`]. - pub(crate) async fn select_first_exit(&mut self) -> FirstExit { - let shutdown_signal = tokio::signal::ctrl_c(); - tokio::pin!(shutdown_signal); - tokio::select! { - signal_result = &mut shutdown_signal => signal_result.into(), - server_result = &mut self.server => server_result.into(), - lane_result = &mut self.lane => lane_result.into(), - reader_result = &mut self.reader => reader_result.into(), - submitter_result = &mut self.submitter => submitter_result.into(), - detector_result = &mut self.detector => detector_result.into(), - fee_oracle_result = async { - match self.fee_oracle.as_mut() { - Some(handle) => Some(handle.await.into()), - None => std::future::pending().await, - } - } => fee_oracle_result.expect("fixed mode does not select an oracle exit"), - } - } - - /// Drive orderly cleanup: request shutdown, await each worker (logging - /// cleanup-time errors), surface the primary failure (or the signal- - /// handler error, which takes priority over component errors observed - /// during shutdown). - pub(crate) async fn finish(self, first_exit: FirstExit) -> Result<(), RunError> { - self.shutdown.request_shutdown(); - - let Self { - server, - lane, - reader, - submitter, - detector, - fee_oracle, - shutdown: _, - } = self; - let mut components: Vec<(WorkerId, ComponentShutdown)> = vec![ - (WorkerId::Server, Box::pin(wait_for_server_shutdown(server))), - (WorkerId::Lane, Box::pin(wait_for_lane_shutdown(lane))), - ( - WorkerId::InputReader, - Box::pin(wait_for_input_reader_shutdown(reader)), - ), - ( - WorkerId::BatchSubmitter, - Box::pin(wait_for_batch_submitter_shutdown(submitter)), - ), - ( - WorkerId::DangerDetector, - Box::pin(wait_for_danger_detector_shutdown(detector)), - ), - ]; - if let Some(fee_oracle) = fee_oracle { - components.push(( - WorkerId::FeeOracle, - Box::pin(wait_for_fee_oracle_shutdown(fee_oracle)), - )); - } - - // Two completion modes: - // - Worker-failure: we already have the primary; await the OTHER - // components for orderly cleanup, log any cleanup errors, surface - // the primary (wrapped to RunError). - // - Signal-driven shutdown: an OS signal triggered shutdown. Wait for - // everything to drain; the signal handler's own error (if any) - // takes priority over any subsequent component shutdown error. - let (worker_failure, signal_error): (Option<(WorkerId, WorkerExit)>, Option) = - match first_exit { - FirstExit::Signal(err) => (None, err), - FirstExit::Worker(exit) => { - let id = exit.worker_id(); - (Some((id, exit)), None) - } - }; - - if let Some((failed, primary_exit)) = worker_failure { - for (id, fut) in components { - if id == failed { - // Drop the primary's future without awaiting — its task - // is already done (it's what tripped the select), and - // we'll surface its error directly below. - drop(fut); - continue; - } - log_cleanup_result(id.label(), fut.await); - } - return Err(RunError::Worker(primary_exit)); - } - - // Signal path: await EVERY component so each worker's JoinHandle is - // joined and its task fully drains. A `break` here would drop the - // remaining components' futures un-awaited, which DETACHES those tasks - // (only `JoinHandle::abort()` cancels a dropped handle) — they'd be - // killed mid-drain at runtime teardown, the exact abrupt-write case the - // startup snapshot hygiene (sweep/gc/re-stamp) exists to clean up after. - // Keep the first error to surface; log every error (the signal handler's - // own error, if any, still takes priority below). - let mut shutdown_error: Option = None; - for (id, fut) in components { - if let Err(e) = fut.await { - warn!(component = id.label(), error = %e, "component errored during signal-driven shutdown"); - if shutdown_error.is_none() { - shutdown_error = Some(e); - } - } - } - match (signal_error, shutdown_error) { - (Some(err), _) => Err(err), - (None, Some(exit)) => Err(RunError::Worker(exit)), - (None, None) => Ok(()), - } - } -} - -/// Stable identity of each long-lived worker. The `finish` worker-failure path -/// skips the already-exited worker by matching on this enum, not on a label -/// string that could silently drift from the component-array order. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -enum WorkerId { - Server, - Lane, - InputReader, - BatchSubmitter, - DangerDetector, - FeeOracle, -} - -impl WorkerId { - /// Human-readable label for logs, matching the `Workers::finish` list. - fn label(self) -> &'static str { - match self { - WorkerId::Server => "server", - WorkerId::Lane => "inclusion lane", - WorkerId::InputReader => "input reader", - WorkerId::BatchSubmitter => "batch submitter", - WorkerId::DangerDetector => "danger detector", - WorkerId::FeeOracle => "fee oracle", - } - } -} - -impl WorkerExit { - /// Which worker produced this exit. - fn worker_id(&self) -> WorkerId { - match self { - WorkerExit::Server(_) => WorkerId::Server, - WorkerExit::Lane(_) => WorkerId::Lane, - WorkerExit::InputReader(_) => WorkerId::InputReader, - WorkerExit::BatchSubmitter(_) => WorkerId::BatchSubmitter, - WorkerExit::DangerDetector(_) => WorkerId::DangerDetector, - WorkerExit::FeeOracle(_) => WorkerId::FeeOracle, - } - } -} - -// ── `From` for FirstExit ────────────────────────────────── -// -// Each `select!` arm awaits a future and converts the result into a -// `FirstExit`. We dispatch via these `From` impls so the select arms read as -// uniform one-liners (`result.into()`); the worker-specific mapping logic -// lives here, with each input type uniquely identifying its worker. - -/// ctrl_c shutdown signal: `Ok(())` = clean signal, `Err(io)` = signal-handler -/// installation failed. -impl From> for FirstExit { - fn from(result: Result<(), std::io::Error>) -> Self { - FirstExit::Signal(result.err().map(RunError::from)) - } -} - -impl From, tokio::task::JoinError>> for FirstExit { - fn from(result: Result, tokio::task::JoinError>) -> Self { - FirstExit::Worker(WorkerExit::Server(match result { - Ok(Ok(())) => ServerExit::StoppedUnexpectedly, - Ok(Err(source)) => ServerExit::Source(source), - Err(source) => ServerExit::Join(source), - })) - } -} - -impl From, tokio::task::JoinError>> for FirstExit { - fn from(result: Result, tokio::task::JoinError>) -> Self { - FirstExit::Worker(WorkerExit::Lane(match result { - Ok(Ok(())) => LaneExit::StoppedUnexpectedly, - Ok(Err(source)) => LaneExit::Source(source), - Err(source) => LaneExit::Join(source), - })) - } -} - -impl From, tokio::task::JoinError>> for FirstExit { - fn from(result: Result, tokio::task::JoinError>) -> Self { - FirstExit::Worker(WorkerExit::InputReader(match result { - Ok(Ok(())) => InputReaderExit::StoppedUnexpectedly, - Ok(Err(source)) => InputReaderExit::Source(source), - Err(source) => InputReaderExit::Join(source), - })) - } -} - -impl From, tokio::task::JoinError>> - for FirstExit -{ - fn from( - result: Result, tokio::task::JoinError>, - ) -> Self { - FirstExit::Worker(WorkerExit::BatchSubmitter(match result { - // Worker returning `Shutdown` outside of a real shutdown means it - // stopped on its own — treat as unexpected. - Ok(Ok(SubmitterExit::Shutdown)) => BatchSubmitterExit::StoppedUnexpectedly, - Ok(Err(source)) => BatchSubmitterExit::Source(source), - Err(source) => BatchSubmitterExit::Join(source), - })) - } -} - -impl From, tokio::task::JoinError>> for FirstExit { - fn from( - result: Result, tokio::task::JoinError>, - ) -> Self { - FirstExit::Worker(WorkerExit::DangerDetector(match result { - // Detector Shutdown means its own shutdown signal fired, which - // only happens after runtime-wide shutdown was triggered. Treat - // as unexpected if it wins the select. - Ok(Ok(DetectorExit::Shutdown)) => DangerDetectorExit::StoppedUnexpectedly, - Ok(Ok(DetectorExit::RecoveryRequired { status })) => { - DangerDetectorExit::DangerDetected { status } - } - Ok(Err(source)) => DangerDetectorExit::Source(source), - Err(source) => DangerDetectorExit::Join(source), - })) - } -} - -impl From, tokio::task::JoinError>> - for FirstExit -{ - fn from( - result: Result< - Result<(), crate::l1::fee_oracle::worker::FeeOracleError>, - tokio::task::JoinError, - >, - ) -> Self { - FirstExit::Worker(WorkerExit::FeeOracle(match result { - Ok(Ok(())) => FeeOracleExit::StoppedUnexpectedly, - Ok(Err(source)) => FeeOracleExit::Source(source), - Err(source) => FeeOracleExit::Join(source), - })) - } -} - -// ── Shutdown waiters ─────────────────────────────────────────────────── -// -// Each waiter awaits a worker's JoinHandle and converts via the per-worker -// `*Exit::from_shutdown` constructor (which knows `Ok(())` is graceful). -// Same shape per worker; kept explicit for readability. - -type ComponentShutdown = Pin> + Send>>; - -async fn wait_for_server_shutdown( - server_task: JoinHandle>, -) -> Result<(), WorkerExit> { - ServerExit::from_shutdown(server_task.await).map_err(Into::into) -} - -async fn wait_for_lane_shutdown( - handle: JoinHandle>, -) -> Result<(), WorkerExit> { - LaneExit::from_shutdown(handle.await).map_err(Into::into) -} - -async fn wait_for_input_reader_shutdown( - handle: JoinHandle>, -) -> Result<(), WorkerExit> { - InputReaderExit::from_shutdown(handle.await).map_err(Into::into) -} - -async fn wait_for_batch_submitter_shutdown( - handle: JoinHandle>, -) -> Result<(), WorkerExit> { - BatchSubmitterExit::from_shutdown(handle.await).map_err(Into::into) -} - -async fn wait_for_danger_detector_shutdown( - handle: JoinHandle>, -) -> Result<(), WorkerExit> { - DangerDetectorExit::from_shutdown(handle.await).map_err(Into::into) -} - -async fn wait_for_fee_oracle_shutdown( - handle: JoinHandle>, -) -> Result<(), WorkerExit> { - FeeOracleExit::from_shutdown(handle.await).map_err(Into::into) -} - -fn log_cleanup_result(component: &str, result: Result<(), WorkerExit>) { - if let Err(err) = result { - warn!(component, error = %err, "component shutdown after primary failure also errored"); - } -} - -// Built once at worker spawn (sync, raw `create_signer_provider`). The submitter -// is long-lived, so a one-shot spawn-time chain-id check would go stale; the -// keyed-write guard instead lives in `EthereumBatchPoster::submit_batches`, -// which re-confirms the chain id immediately before every productive send. -fn build_batch_submitter_provider(l1: &L1Config) -> Result { - crate::l1::provider::create_signer_provider( - &l1.eth_rpc_url, - &l1.batch_submitter_private_key, - l1.allow_insecure_rpc, - ) - .map_err(std::io::Error::other) -} - -/// Require the finalized snapshot the lane will `from_dump` against. `setup` -/// registers the genesis snapshot and `run` gates on the `setup_complete` -/// marker, so by the time the lane starts the snapshot must exist. A missing -/// one means the DB's setup is incomplete/corrupt — terminal -/// `SetupNotComplete` (re-run `setup`), not a cold-start to silently heal. -fn require_finalized_snapshot(storage: &mut crate::storage::Storage) -> Result<(), RunError> { - if storage.finalized_dump()?.is_none() { - return Err(RunError::Bootstrap( - crate::runtime::error::BootstrapError::SetupNotComplete, - )); - } - Ok(()) -} - -/// Re-stamp `B` into the finalized dump's `info.toml` from the -/// authoritative DB row. Idempotent; closes the crash window between a -/// promotion's commit and the lane's in-place stamp. -fn restamp_finalized_promotion(storage: &mut crate::storage::Storage) -> Result<(), RunError> { - if let Some(finalized) = storage.finalized_dump()? { - dump_info::stamp_promoted_inclusion_block( - &finalized.dump.prefix, - finalized.inclusion_block, - )?; - } - Ok(()) -} - -/// Drop any dump rows that are now unreferenced (no pending, no -/// finalized, no leases). The companion `sweep_orphan_dumps` then -/// catches anything on disk that this leaves behind, plus -/// crash-during-create_dump orphans the SQLite layer never saw. -fn snapshot_gc_at_startup( - storage: &mut crate::storage::Storage, -) -> Result { - let removed = storage.gc_unreferenced_dumps()?; - for row in &removed { - if let Err(err) = delete_dump_dir::(&row.prefix) { - tracing::warn!( - error = %err, - prefix = ?row.prefix, - "startup GC: filesystem delete failed; orphan left for sweep", - ); - } - } - Ok(removed.len()) -} - -/// Walk `dumps_dir` and delete any dump directory that isn't in -/// `Storage::list_dump_rows`. Catches: -/// -/// - **crash-during-create**: a dump dir exists on disk (possibly -/// without its app subtree or `info.toml`) but no SQLite row was -/// ever written for it. -/// - **crash-during-GC**: SQLite row was deleted but the filesystem -/// delete either wasn't reached or failed. -/// -/// Filesystem-only — no SQLite writes here. Failures log and -/// continue (the next startup retries). The post-`ensure_finalized` -/// ordering matters: the genesis dump's dir is in -/// `list_dump_rows` by the time this runs, so we never delete it. -fn sweep_orphan_dumps( - storage: &mut crate::storage::Storage, - dumps_dir: &std::path::Path, -) -> Result { - let known: std::collections::HashSet = storage - .list_dump_rows()? - .into_iter() - .map(|row| row.prefix) - .collect(); - let mut removed = 0; - for entry in std::fs::read_dir(dumps_dir)? { - let entry = entry?; - let path = entry.path(); - if known.contains(&path) { - continue; - } - match delete_dump_dir::(&path) { - Ok(()) => removed += 1, - Err(err) => { - tracing::warn!( - error = %err, - ?path, - "orphan dump sweep: delete failed; will retry next startup", - ); - } - } - } - Ok(removed) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::recovery::DangerDetectorError; - use crate::storage::DangerStatus; - - // ── select!-arm `From` conversions ────────────────── - // - // The detector arm is the interesting one (DangerDetected vs Shutdown vs - // Source vs Join). The other workers follow a uniform 3-way mapping - // covered by the type system. - - type DetectorJoinResult = - Result, tokio::task::JoinError>; - - #[test] - fn detector_shutdown_in_select_maps_to_stopped_unexpectedly() { - let result: DetectorJoinResult = Ok(Ok(DetectorExit::Shutdown)); - assert!(matches!( - FirstExit::from(result), - FirstExit::Worker(WorkerExit::DangerDetector( - DangerDetectorExit::StoppedUnexpectedly - )) - )); - } - - #[test] - fn detector_recovery_required_maps_to_danger_detected() { - let result: DetectorJoinResult = Ok(Ok(DetectorExit::RecoveryRequired { - status: DangerStatus::ClosedBatchInDanger(7), - })); - assert!(matches!( - FirstExit::from(result), - FirstExit::Worker(WorkerExit::DangerDetector( - DangerDetectorExit::DangerDetected { - status: DangerStatus::ClosedBatchInDanger(7) - } - )) - )); - } - - // ── Snapshot startup hygiene ──────────────────────────────────── - // - // `sweep_orphan_dumps` removes filesystem entries under `dumps_dir` - // that aren't tracked in SQLite. Catches crash-during-create_dump - // (file exists, no row) and crash-during-GC (row deleted, file - // remains). Must NOT touch directories that ARE in `dumps` — - // those are the genesis dump, finalized, and any pending the - // lane is still working with. - - use crate::runtime::test_support::{SweepTestApp, create_structured_dump}; - use crate::storage::Storage; - use crate::storage::test_helpers::temp_db; - - #[test] - fn sweep_orphan_dumps_removes_directories_not_in_storage() { - let db = temp_db("sweep-orphans"); - let mut storage = Storage::open(db.path.as_str()).expect("open"); - let dumps_dir = tempfile::tempdir().expect("dumps dir"); - - // Tracked dump (in SQLite). - let tracked = dumps_dir.path().join("tracked"); - create_structured_dump(&tracked); - storage - .insert_finalized_dump(&tracked, 0, 0) - .expect("register tracked"); - - // Two orphans (NOT in SQLite). One is fully formed; the other - // mimics a crash between dir creation and the app dump (no - // `state` subtree) — the sweep must remove both. - let orphan_a = dumps_dir.path().join("orphan-a"); - let orphan_b = dumps_dir.path().join("orphan-b"); - create_structured_dump(&orphan_a); - std::fs::create_dir(&orphan_b).expect("orphan b dir"); - - let removed = sweep_orphan_dumps::(&mut storage, dumps_dir.path()).unwrap(); - assert_eq!(removed, 2); - assert!(tracked.exists(), "tracked dump must survive"); - assert!(!orphan_a.exists()); - assert!(!orphan_b.exists()); - } - - #[test] - fn sweep_orphan_dumps_on_empty_directory_is_noop() { - let db = temp_db("sweep-empty"); - let mut storage = Storage::open(db.path.as_str()).expect("open"); - let dumps_dir = tempfile::tempdir().expect("dumps dir"); - - let removed = sweep_orphan_dumps::(&mut storage, dumps_dir.path()).unwrap(); - assert_eq!(removed, 0); - } - - #[test] - fn snapshot_gc_at_startup_removes_unreferenced_rows() { - let db = temp_db("gc-startup"); - let mut storage = Storage::open(db.path.as_str()).expect("open"); - let dumps_dir = tempfile::tempdir().expect("dumps dir"); - - // Two dumps: superseded + finalized. - let superseded = dumps_dir.path().join("superseded"); - let finalized = dumps_dir.path().join("finalized"); - create_structured_dump(&superseded); - create_structured_dump(&finalized); - storage - .insert_pending_dump(&superseded, 0, 0) - .expect("pending 0"); - storage.promote_finalized(0, 0).expect("promote 0"); - storage - .insert_pending_dump(&finalized, 1, 0) - .expect("pending 1"); - storage.promote_finalized(1, 0).expect("promote 1"); - // `superseded`'s row is now unreferenced (replaced by - // finalized's promotion), but the directory is still on disk. - - let removed = snapshot_gc_at_startup::(&mut storage).unwrap(); - assert_eq!(removed, 1); - assert!(!superseded.exists(), "GC removed the superseded directory"); - assert!(finalized.exists(), "current finalized survived"); - } - - #[test] - fn detector_inner_error_maps_to_source_variant() { - let result: DetectorJoinResult = Ok(Err(DangerDetectorError::Join("boom".into()))); - assert!(matches!( - FirstExit::from(result), - FirstExit::Worker(WorkerExit::DangerDetector(DangerDetectorExit::Source(_))) - )); - } -} diff --git a/sequencer/src/storage/admin.rs b/sequencer/src/storage/admin.rs index 5c069e7c..539e7bd5 100644 --- a/sequencer/src/storage/admin.rs +++ b/sequencer/src/storage/admin.rs @@ -38,6 +38,75 @@ impl Storage { mod tests { use crate::storage::{Storage, test_helpers::temp_db}; + #[test] + fn negative_batch_size_target_floors_to_zero() { + let db = temp_db("floor-batch-target"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + + // This row satisfies every schema constraint while deriving a negative + // target: 1403 - 2000 - 419 = -1016. + storage + .conn + .execute( + "UPDATE batch_policy SET log_alpha = 2000 WHERE singleton_id = 0", + [], + ) + .expect("set high alpha"); + + let policy = storage.batch_policy().expect("read policy"); + assert_eq!( + policy.batch_size_target, 0, + "negative batch-size target should be floored to zero" + ); + } + + #[test] + #[should_panic(expected = "batch policy derived recommended fee")] + fn negative_recommended_fee_fails_loud() { + let db = temp_db("negative-recommended-fee"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + + // A negative derived fee is ruled out by the column constraints. Model + // corruption/tampering that bypassed them and ensure the read cannot + // launder the impossible value into a plausible zero fee. + storage + .conn + .execute_batch( + "PRAGMA ignore_check_constraints = ON; + UPDATE batch_policy SET log_delta = -10000 WHERE singleton_id = 0;", + ) + .expect("inject impossible policy row"); + + let _ = storage.batch_policy(); + } + + #[test] + #[should_panic(expected = "batch policy derived batch size target")] + fn oversized_batch_size_target_fails_loud() { + let db = temp_db("oversized-batch-target"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + + // This is schema-valid: the derived target is 17984, which remains + // below the row's configured log_max_batch_bytes of 20000. It is still + // outside the core fee exponent domain and must not become u64::MAX. + storage + .conn + .execute( + "UPDATE batch_policy + SET log_alpha = -17000, log_max_batch_bytes = 20000 + WHERE singleton_id = 0", + [], + ) + .expect("set schema-valid oversized target"); + let integrity: String = storage + .conn + .query_row("PRAGMA integrity_check", [], |row| row.get(0)) + .expect("check database integrity"); + assert_eq!(integrity, "ok"); + + let _ = storage.batch_policy(); + } + #[test] #[should_panic(expected = "num + denom overflows u64")] fn set_alpha_rejects_overflow() { diff --git a/sequencer/src/storage/convert.rs b/sequencer/src/storage/convert.rs index be26c289..e8cb9536 100644 --- a/sequencer/src/storage/convert.rs +++ b/sequencer/src/storage/convert.rs @@ -1,21 +1,44 @@ // (c) Cartesi and individual authors (see AUTHORS) // SPDX-License-Identifier: Apache-2.0 (see LICENSE) -//! Saturating width conversions between Rust and SQLite integer types, plus -//! `SystemTime` ↔ `i64` Unix-ms conversions. +//! Integer conversions between Rust domain types and SQLite `INTEGER` columns, +//! plus `SystemTime` ↔ `i64` Unix-ms conversions. //! -//! SQLite stores integers as `INTEGER` (signed 64-bit). Rust domain types use -//! narrower unsigned widths (`u16`, `u32`, `u64`). The conversions here are -//! load-bearing glue that the rest of the storage module calls pervasively. +//! Two conversion families with opposite postures, per the fail-loud check +//! policy in `docs/invariants.md`: //! -//! All conversions saturate rather than panic — the domain values we persist -//! are always non-negative and well within `i64::MAX`, but saturation keeps -//! corrupted or malicious DB rows from crashing the process. +//! - **Contract-bound conversions fail loud.** Domain values (indices, nonces, +//! block numbers, offsets, fees) are non-negative and far below `i64::MAX` +//! by schema `CHECK`s, triggers, or writer-side types. An out-of-range value +//! can only mean DB corruption, tampering, or a sequencer bug; saturating it +//! would fabricate a plausible value and let the divergence externalize (a +//! signed batch, a feed event, a wrong recovery pivot). These panic to stop +//! the operation before externalization. A persistent violation is not +//! repaired by restart and may require inspection or cockroach recovery; +//! fail-loud is a safety property, not a self-healing claim. +//! - **Clock and query-bound conversions saturate.** Wall-clock time is +//! environmental, not an invariant: a far-future clock clamps to +//! `i64::MAX` rather than aborting. Untrusted or config-sourced SQL bounds +//! go through [`saturating_query_bound`], where clamping preserves the +//! comparison semantics exactly. Sign remains contract-bound even for clock +//! *columns*: every timestamp writer is u64-clock-sourced (floored at 0), so +//! [`from_unix_ms`] fail-louds on a negative stored value — a sign check is +//! a real invariant, unlike cross-column timestamp ordering checks, which +//! once wedged recovery and are deliberately left unenforced. use std::time::{Duration, SystemTime, UNIX_EPOCH}; +#[derive(Debug, thiserror::Error)] +#[error("{field} value {value} exceeds SQLite INTEGER maximum")] +struct ExternalIntegerRangeError { + field: &'static str, + value: u64, +} + // ── Time helpers ────────────────────────────────────────────────────────── +/// Saturating: a pre-epoch clock clamps to 0, a far-future clock to +/// `i64::MAX`. Wall-clock state is environmental, never an invariant. pub(super) fn to_unix_ms(time: SystemTime) -> i64 { time.duration_since(UNIX_EPOCH) .unwrap_or_default() @@ -24,37 +47,315 @@ pub(super) fn to_unix_ms(time: SystemTime) -> i64 { .unwrap_or(i64::MAX) } +/// Fail-loud on sign: every timestamp writer is u64-clock-sourced, so a +/// negative stored Unix-ms value is contract-impossible. Ordering and +/// monotonicity are deliberately NOT checked here (wall-clock regression is +/// legitimate). pub(super) fn from_unix_ms(ms: i64) -> SystemTime { - let clamped_ms = ms.max(0) as u64; - UNIX_EPOCH + Duration::from_millis(clamped_ms) + let ms = u64::try_from(ms).unwrap_or_else(|_| { + panic!("stored unix-ms timestamp {ms} is negative: contract-impossible") + }); + UNIX_EPOCH + Duration::from_millis(ms) } -/// Current wall-clock time as an `i64` SQLite timestamp. +/// Current wall-clock time as an `i64` SQLite timestamp. Saturating (see +/// [`to_unix_ms`]). /// -/// Delegates to [`crate::runtime::clock::unix_now_ms`] so the whole crate goes +/// Delegates to [`crate::clock::unix_now_ms`] so the whole crate goes /// through one clock entry point. pub(super) fn now_unix_ms() -> i64 { - i64::try_from(crate::runtime::clock::unix_now_ms()).unwrap_or(i64::MAX) + i64::try_from(crate::clock::unix_now_ms()).unwrap_or(i64::MAX) } -// ── Width conversions ───────────────────────────────────────────────────── +// ── Contract-bound width conversions (fail loud) ────────────────────────── pub(super) fn u64_to_i64(value: u64) -> i64 { - i64::try_from(value).unwrap_or(i64::MAX) + i64::try_from(value) + .unwrap_or_else(|_| panic!("domain value {value} exceeds i64::MAX: contract-impossible")) } -pub(super) fn usize_to_i64(value: usize) -> i64 { - i64::try_from(value).unwrap_or(i64::MAX) +/// Checked conversion for configuration/provider values crossing into the +/// SQLite representation. An unrepresentable external value is a typed +/// boundary refusal, not an internal invariant panic. +pub(super) fn external_u64_to_i64(value: u64, field: &'static str) -> rusqlite::Result { + i64::try_from(value).map_err(|_| { + rusqlite::Error::ToSqlConversionFailure(Box::new(ExternalIntegerRangeError { + field, + value, + })) + }) } pub(super) fn i64_to_u64(value: i64) -> u64 { - value.max(0) as u64 + u64::try_from(value) + .unwrap_or_else(|_| panic!("stored value {value} is negative: contract-impossible")) } pub(super) fn i64_to_u16(value: i64) -> u16 { - u16::try_from(value.max(0)).unwrap_or(u16::MAX) + u16::try_from(value).unwrap_or_else(|_| { + panic!("stored value {value} is outside u16 range: contract-impossible") + }) } pub(super) fn i64_to_u32(value: i64) -> u32 { - u32::try_from(value.max(0)).unwrap_or(u32::MAX) + u32::try_from(value).unwrap_or_else(|_| { + panic!("stored value {value} is outside u32 range: contract-impossible") + }) +} + +// ── Query-bound conversion (saturating, by design) ──────────────────────── + +/// Saturating clamp for **untrusted or config-sourced** SQL query bounds: +/// WS `from_offset` cursors, page/count `LIMIT`s, and setup/recovery block +/// predicates. The full `u64` range is legal input here, and clamping to +/// `i64::MAX` preserves the comparison exactly — no SQLite `INTEGER` or rowid +/// exceeds `i64::MAX`, so a past-the-end lower bound matches zero rows while a +/// clamped upper bound or `LIMIT` includes every representable row. +/// +/// Never use this for domain values read from or written to columns; those +/// go through the fail-loud converters above. +pub(super) fn saturating_query_bound(value: u64) -> i64 { + i64::try_from(value).unwrap_or(i64::MAX) +} + +/// Whether a SQLite error proves durable row/schema corruption or a +/// trusted-code contract violation, as opposed to an operational condition +/// such as BUSY, I/O failure, permissions, or disk pressure. +/// +/// Egress uses this distinction to take the whole runtime offline on +/// persistent corruption without terminalizing transient database failures. +/// +/// The trailing wildcard is forced (`rusqlite::Error` is non-exhaustive) and +/// deliberately fail-open toward *operational*: an unknown/new variant +/// restarts rather than terminalizes, because a wrong "terminal" pages an +/// operator for a self-healing condition while a wrong "operational" merely +/// retries into the same error until it is classified. Review the list on +/// every rusqlite upgrade. +pub(crate) fn is_persistent_storage_error(error: &rusqlite::Error) -> bool { + use rusqlite::Error; + use rusqlite::ffi::ErrorCode; + + match error { + Error::SqliteFailure(source, _) => matches!( + source.code, + ErrorCode::InternalMalfunction + | ErrorCode::DatabaseCorrupt + | ErrorCode::SchemaChanged + | ErrorCode::ConstraintViolation + | ErrorCode::TypeMismatch + | ErrorCode::ApiMisuse + | ErrorCode::NotADatabase + | ErrorCode::Unknown + ), + Error::FromSqlConversionFailure(..) + | Error::IntegralValueOutOfRange(..) + | Error::Utf8Error(..) + | Error::NulError(..) + | Error::InvalidParameterName(..) + | Error::ExecuteReturnedResults + | Error::QueryReturnedNoRows + | Error::QueryReturnedMoreThanOneRow + | Error::InvalidColumnIndex(..) + | Error::InvalidColumnName(..) + | Error::InvalidColumnType(..) + | Error::StatementChangedRows(..) + | Error::ToSqlConversionFailure(..) + | Error::InvalidQuery + | Error::UnwindingPanic + | Error::MultipleStatement + | Error::InvalidParameterCount(..) => true, + // The modern_sqlite (bundled) build reports offset-bearing + // prepare-time failures (malformed SQL) as SqlInputError; the same + // trusted-SQL-broke condition without an offset ("no such table") + // arrives as SqliteFailure(Unknown). Both spellings must classify + // identically. + Error::SqlInputError { .. } => true, + _ => false, + } +} + +#[cfg(test)] +mod tests { + use super::{ + external_u64_to_i64, from_unix_ms, i64_to_u16, i64_to_u32, i64_to_u64, + is_persistent_storage_error, saturating_query_bound, to_unix_ms, u64_to_i64, + }; + use std::time::{Duration, UNIX_EPOCH}; + + const I64_MAX_U64: u64 = i64::MAX as u64; + + #[test] + fn prepare_time_sql_failures_classify_persistent_in_both_spellings() { + // Trusted SQL breaking at prepare time has two rusqlite spellings in + // the bundled (modern_sqlite) build: offset-bearing malformed SQL is + // SqlInputError, while "no such table" (no offset) stays + // SqliteFailure(Unknown). Both must classify persistent. + let conn = rusqlite::Connection::open_in_memory().expect("open in-memory db"); + + let err = conn + .prepare("SELECT 1 FRO somewhere") + .expect_err("prepare of malformed SQL must fail"); + assert!( + matches!(err, rusqlite::Error::SqlInputError { .. }), + "expected SqlInputError from an offset-bearing prepare failure, got {err:?}" + ); + assert!(is_persistent_storage_error(&err)); + + let err = conn + .prepare("SELECT value FROM definitely_missing_table") + .expect_err("prepare against a missing table must fail"); + assert!( + matches!( + &err, + rusqlite::Error::SqliteFailure(source, _) + if source.code == rusqlite::ffi::ErrorCode::Unknown + ), + "expected SqliteFailure(Unknown) from a missing table, got {err:?}" + ); + assert!(is_persistent_storage_error(&err)); + } + + #[test] + fn unix_ms_conversion_saturates_environmental_time_boundaries() { + assert_eq!( + to_unix_ms(UNIX_EPOCH - Duration::from_millis(1)), + 0, + "pre-epoch time floors at zero" + ); + assert_eq!(to_unix_ms(UNIX_EPOCH), 0); + assert_eq!( + to_unix_ms(UNIX_EPOCH + Duration::from_millis(I64_MAX_U64)), + i64::MAX + ); + assert_eq!( + to_unix_ms(UNIX_EPOCH + Duration::from_millis(I64_MAX_U64 + 1)), + i64::MAX, + "far-future time caps at SQLite's maximum INTEGER" + ); + } + + #[test] + fn stored_unix_ms_accepts_full_non_negative_i64_range() { + assert_eq!(from_unix_ms(0), UNIX_EPOCH); + assert_eq!( + from_unix_ms(i64::MAX), + UNIX_EPOCH + Duration::from_millis(I64_MAX_U64) + ); + } + + #[test] + #[should_panic(expected = "stored unix-ms timestamp -1 is negative: contract-impossible")] + fn stored_unix_ms_rejects_negative_values() { + let _ = from_unix_ms(-1); + } + + #[test] + fn u64_to_i64_accepts_representable_boundaries() { + assert_eq!(u64_to_i64(0), 0); + assert_eq!(u64_to_i64(I64_MAX_U64), i64::MAX); + } + + #[test] + #[should_panic(expected = "exceeds i64::MAX: contract-impossible")] + fn u64_to_i64_rejects_first_unrepresentable_value() { + let _ = u64_to_i64(I64_MAX_U64 + 1); + } + + #[test] + fn external_u64_to_i64_returns_a_typed_boundary_error() { + assert_eq!( + external_u64_to_i64(I64_MAX_U64, "test field").unwrap(), + i64::MAX + ); + let err = external_u64_to_i64(I64_MAX_U64 + 1, "test field") + .expect_err("external value must be rejected"); + assert!( + matches!(err, rusqlite::Error::ToSqlConversionFailure(_)), + "unexpected error: {err}" + ); + assert!(err.to_string().contains("test field")); + } + + #[test] + fn i64_to_u64_accepts_non_negative_boundaries() { + assert_eq!(i64_to_u64(0), 0); + assert_eq!(i64_to_u64(i64::MAX), I64_MAX_U64); + } + + #[test] + #[should_panic(expected = "stored value -1 is negative: contract-impossible")] + fn i64_to_u64_rejects_negative_values() { + let _ = i64_to_u64(-1); + } + + #[test] + fn i64_to_u16_accepts_unsigned_boundaries() { + assert_eq!(i64_to_u16(0), 0); + assert_eq!(i64_to_u16(i64::from(u16::MAX)), u16::MAX); + } + + #[test] + #[should_panic(expected = "outside u16 range: contract-impossible")] + fn i64_to_u16_rejects_negative_values() { + let _ = i64_to_u16(-1); + } + + #[test] + #[should_panic(expected = "outside u16 range: contract-impossible")] + fn i64_to_u16_rejects_first_value_above_range() { + let _ = i64_to_u16(i64::from(u16::MAX) + 1); + } + + #[test] + fn i64_to_u32_accepts_unsigned_boundaries() { + assert_eq!(i64_to_u32(0), 0); + assert_eq!(i64_to_u32(i64::from(u32::MAX)), u32::MAX); + } + + #[test] + #[should_panic(expected = "outside u32 range: contract-impossible")] + fn i64_to_u32_rejects_negative_values() { + let _ = i64_to_u32(-1); + } + + #[test] + #[should_panic(expected = "outside u32 range: contract-impossible")] + fn i64_to_u32_rejects_first_value_above_range() { + let _ = i64_to_u32(i64::from(u32::MAX) + 1); + } + + #[test] + fn query_bound_saturates_only_above_sqlite_integer_range() { + assert_eq!(saturating_query_bound(0), 0); + assert_eq!(saturating_query_bound(I64_MAX_U64), i64::MAX); + assert_eq!( + saturating_query_bound(I64_MAX_U64 + 1), + i64::MAX, + "first out-of-range bound caps" + ); + assert_eq!( + saturating_query_bound(u64::MAX), + i64::MAX, + "full input range is legal" + ); + } + + #[test] + fn persistent_storage_error_classifier_separates_corruption_from_io() { + assert!(is_persistent_storage_error( + &rusqlite::Error::QueryReturnedNoRows + )); + assert!(is_persistent_storage_error( + &rusqlite::Error::InvalidColumnIndex(7) + )); + assert!(!is_persistent_storage_error( + &rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code: rusqlite::ffi::ErrorCode::DatabaseBusy, + extended_code: 5, + }, + None, + ) + )); + } } diff --git a/sequencer/src/storage/egress.rs b/sequencer/src/storage/egress.rs index d2692e6c..71d20a05 100644 --- a/sequencer/src/storage/egress.rs +++ b/sequencer/src/storage/egress.rs @@ -11,8 +11,9 @@ use alloy_primitives::{Address, B256}; use rusqlite::{Result, params}; use super::Storage; -use super::convert::{i64_to_u64, u64_to_i64, usize_to_i64}; +use super::convert::{i64_to_u32, i64_to_u64, saturating_query_bound}; use super::queries::decode_l2_tx_row; +use sequencer_core::history::ExecutedInputCount; use sequencer_core::l2_tx::{DirectInput, SequencedL2Tx, ValidUserOp}; /// One persisted L2 transaction and the ordering context of its covering frame. @@ -24,6 +25,7 @@ pub(crate) enum OrderedL2TxRow { nonce: u32, safe_block: u64, batch_nonce: u64, + executed_input_offset: Option, }, DirectInput { offset: u64, @@ -33,6 +35,7 @@ pub(crate) enum OrderedL2TxRow { batch_nonce: u64, block_timestamp: u64, transaction_hash: B256, + executed_input_offset: Option, }, } @@ -43,41 +46,66 @@ impl OrderedL2TxRow { } } - fn into_replay_parts(self) -> (u64, SequencedL2Tx, u64) { + fn into_replay_row(self) -> ReplayL2TxRow { match self { Self::UserOp { offset, tx, safe_block, + executed_input_offset, .. - } => (offset, SequencedL2Tx::UserOp(tx), safe_block), + } => ReplayL2TxRow { + db_offset: offset, + tx: SequencedL2Tx::UserOp(tx), + frame_safe_block: safe_block, + executed_input_offset, + }, Self::DirectInput { offset, tx, safe_block, + executed_input_offset, .. - } => (offset, SequencedL2Tx::Direct(tx), safe_block), + } => ReplayL2TxRow { + db_offset: offset, + tx: SequencedL2Tx::Direct(tx), + frame_safe_block: safe_block, + executed_input_offset, + }, } } } +/// One valid physical replay row with its canonical application attribution. +/// +/// The physical SQLite cursor and logical application offset are deliberately +/// named: they are different coordinates and callers must not infer their +/// meaning from tuple position. +#[derive(Debug, Clone)] +pub(crate) struct ReplayL2TxRow { + pub(crate) db_offset: u64, + pub(crate) tx: SequencedL2Tx, + pub(crate) frame_safe_block: u64, + pub(crate) executed_input_offset: Option, +} + impl Storage { /// Load a page of ordered L2 transactions starting after the given offset. - /// Returns `(db_offset, tx, frame_safe_block)` triples — the third element - /// is the covering frame's `safe_block`. Catch-up replay feeds it to - /// `execute_valid_user_op` so the app's safe-block clock advances exactly - /// as it did live (directs use their own `block_number` instead). Callers - /// should track `db_offset` of the last item as their cursor, not increment - /// a counter. - pub fn ordered_l2_txs_page_from( + /// Each row names both its physical database cursor and optional logical + /// application offset. `frame_safe_block` is fed to user-op replay so the + /// app clock advances exactly as it did live (directs use their own block + /// number). `executed_input_offset` is `None` only for a physical row that + /// does not execute in the application. Callers advance with `db_offset` + /// rather than incrementing either coordinate. + pub(crate) fn ordered_l2_txs_page_from( &mut self, offset: u64, limit: usize, - ) -> Result> { + ) -> Result> { self.ordered_l2_tx_rows_page_from(offset, limit) .map(|rows| { rows.into_iter() - .map(OrderedL2TxRow::into_replay_parts) + .map(OrderedL2TxRow::into_replay_row) .collect() }) } @@ -110,7 +138,8 @@ impl Storage { s.safe_input_index, CASE WHEN s.user_op_pos_in_frame IS NOT NULL THEN u.nonce ELSE NULL END AS op_nonce, CASE WHEN s.safe_input_index IS NOT NULL THEN d.block_timestamp ELSE NULL END AS block_timestamp, - CASE WHEN s.safe_input_index IS NOT NULL THEN d.transaction_hash ELSE NULL END AS transaction_hash + CASE WHEN s.safe_input_index IS NOT NULL THEN d.transaction_hash ELSE NULL END AS transaction_hash, + e.executed_input_offset FROM valid_sequenced_l2_txs s LEFT JOIN user_ops u ON u.batch_index = s.batch_index @@ -123,45 +152,60 @@ impl Storage { ON d.safe_input_index = s.safe_input_index LEFT JOIN batches b ON b.batch_index = s.batch_index + LEFT JOIN executed_inputs e + ON e.sequenced_l2_tx_offset = s.offset WHERE s.offset > ?1 ORDER BY s.offset ASC LIMIT ?2 "; let mut stmt = self.conn.prepare_cached(SQL)?; - let rows = stmt.query_map(params![u64_to_i64(offset), usize_to_i64(limit)], |row| { - let db_offset: i64 = row.get(0)?; - let tx = decode_l2_tx_row( - row.get(1)?, - row.get(2)?, - row.get(3)?, - row.get(4)?, - row.get(5)?, - row.get(6)?, - ); - // Non-NULL for every sequenced row: batches and frames exist before - // anything can be sequenced into them. - let safe_block = i64_to_u64(row.get(7)?); - let batch_nonce = i64_to_u64(row.get(8)?); - match tx { - SequencedL2Tx::UserOp(tx) => Ok(OrderedL2TxRow::UserOp { - offset: i64_to_u64(db_offset), - tx, - nonce: u32::try_from(row.get::<_, i64>(10)?) - .expect("persisted user op nonce must fit u32"), - safe_block, - batch_nonce, - }), - SequencedL2Tx::Direct(tx) => Ok(OrderedL2TxRow::DirectInput { - offset: i64_to_u64(db_offset), - tx, - input_index: i64_to_u64(row.get(9)?), - safe_block, - batch_nonce, - block_timestamp: i64_to_u64(row.get(11)?), - transaction_hash: B256::from_slice(row.get::<_, Vec>(12)?.as_slice()), - }), - } - })?; + let limit = u64::try_from(limit).unwrap_or(u64::MAX); + // Query bounds saturate by design: `offset` can be a client-supplied + // WS cursor and `limit` is config-sourced (see `saturating_query_bound`). + let rows = stmt.query_map( + params![ + saturating_query_bound(offset), + saturating_query_bound(limit) + ], + |row| { + let db_offset: i64 = row.get(0)?; + let tx = decode_l2_tx_row( + row.get(1)?, + row.get(2)?, + row.get(3)?, + row.get(4)?, + row.get(5)?, + row.get(6)?, + ); + // Non-NULL for every sequenced row: batches and frames exist before + // anything can be sequenced into them. + let safe_block = i64_to_u64(row.get(7)?); + let batch_nonce = i64_to_u64(row.get(8)?); + let executed_input_offset = row + .get::<_, Option>(13)? + .map(|value| ExecutedInputCount::new(i64_to_u64(value))); + match tx { + SequencedL2Tx::UserOp(tx) => Ok(OrderedL2TxRow::UserOp { + offset: i64_to_u64(db_offset), + tx, + nonce: i64_to_u32(row.get(10)?), + safe_block, + batch_nonce, + executed_input_offset, + }), + SequencedL2Tx::Direct(tx) => Ok(OrderedL2TxRow::DirectInput { + offset: i64_to_u64(db_offset), + tx, + input_index: i64_to_u64(row.get(9)?), + safe_block, + batch_nonce, + block_timestamp: i64_to_u64(row.get(11)?), + transaction_hash: B256::from_slice(row.get::<_, Vec>(12)?.as_slice()), + executed_input_offset, + }), + } + }, + )?; rows.collect::>>() } @@ -175,51 +219,38 @@ impl Storage { /// Count broadcastable events with offset > `from_offset`, capped at `limit`. /// - /// Used for catch-up window checks. Excludes batch-submitter direct inputs - /// (which are filtered before WS delivery) so the count reflects what the - /// client actually receives. + /// Used for catch-up window checks. Excludes batch-submitter direct + /// inputs — they are filtered before WS delivery, so the count reflects + /// what the client actually receives. pub fn count_broadcastable_events_after( &mut self, from_offset: u64, limit: u64, - batch_submitter_address: Option

, + batch_submitter_address: Address, ) -> Result { if limit == 0 { return Ok(0); } - let value: i64 = match batch_submitter_address { - Some(addr) => { - const SQL: &str = " - SELECT COUNT(*) FROM ( - SELECT 1 FROM valid_sequenced_l2_txs s - WHERE s.offset > ?1 - AND NOT (s.safe_input_index IS NOT NULL - AND EXISTS (SELECT 1 FROM safe_inputs si - WHERE si.safe_input_index = s.safe_input_index - AND si.sender = ?2)) - LIMIT ?3 - )"; - self.conn.query_row( - SQL, - params![u64_to_i64(from_offset), addr.as_slice(), u64_to_i64(limit)], - |row| row.get(0), - )? - } - None => { - const SQL: &str = " - SELECT COUNT(*) FROM ( - SELECT 1 FROM valid_sequenced_l2_txs - WHERE offset > ?1 - LIMIT ?2 - )"; - self.conn.query_row( - SQL, - params![u64_to_i64(from_offset), u64_to_i64(limit)], - |row| row.get(0), - )? - } - }; + const SQL: &str = " + SELECT COUNT(*) FROM ( + SELECT 1 FROM valid_sequenced_l2_txs s + WHERE s.offset > ?1 + AND NOT (s.safe_input_index IS NOT NULL + AND EXISTS (SELECT 1 FROM safe_inputs si + WHERE si.safe_input_index = s.safe_input_index + AND si.sender = ?2)) + LIMIT ?3 + )"; + let value: i64 = self.conn.query_row( + SQL, + params![ + saturating_query_bound(from_offset), + batch_submitter_address.as_slice(), + saturating_query_bound(limit) + ], + |row| row.get(0), + )?; Ok(i64_to_u64(value)) } } diff --git a/sequencer/src/storage/fee_oracle.rs b/sequencer/src/storage/fee_oracle.rs index 8933e206..12c017c8 100644 --- a/sequencer/src/storage/fee_oracle.rs +++ b/sequencer/src/storage/fee_oracle.rs @@ -18,7 +18,7 @@ impl Storage { } /// Unix-ms of the last successful `log_gas_price` write. `0` means never - /// written (migration default); Uniswap treats that as stale. + /// written (migration default); completed setup guarantees a nonzero value. pub fn log_gas_price_updated_at_ms(&self) -> Result { self.conn.query_row( "SELECT log_gas_price_updated_at_ms FROM batch_policy WHERE singleton_id = 0", @@ -27,20 +27,6 @@ impl Storage { ) } - /// Age of the persisted fee-oracle price relative to `now_ms`. - pub fn log_gas_price_age_ms(&self, now_ms: u64) -> Result { - Ok(now_ms.saturating_sub(self.log_gas_price_updated_at_ms()?)) - } - - /// True when the price was never written or is older than `max_age_ms`. - pub fn log_gas_price_is_stale(&self, now_ms: u64, max_age_ms: u64) -> Result { - let updated_at = self.log_gas_price_updated_at_ms()?; - if updated_at == 0 { - return Ok(true); - } - Ok(now_ms.saturating_sub(updated_at) > max_age_ms) - } - pub fn set_log_gas_price(&mut self, log_gas_price: u16) -> Result<()> { let changed = self.conn.execute( "UPDATE batch_policy \ @@ -54,7 +40,7 @@ impl Storage { Ok(()) } - /// Test-only: pin the freshness stamp without going through wall-clock now. + /// Test-only: pin the observation stamp without going through wall-clock now. #[cfg(test)] pub fn set_log_gas_price_updated_at_ms_for_test(&mut self, updated_at_ms: u64) -> Result<()> { use super::convert::u64_to_i64; @@ -91,20 +77,9 @@ mod tests { let db = temp_db("fee-price-stamp"); let mut storage = Storage::open(db.path.as_str()).expect("open storage"); assert_eq!(storage.log_gas_price_updated_at_ms().unwrap(), 0); - assert!(storage.log_gas_price_is_stale(1_000, 100).unwrap()); storage.set_log_gas_price(42).expect("set"); let updated_at = storage.log_gas_price_updated_at_ms().unwrap(); assert!(updated_at > 0); - assert!( - !storage - .log_gas_price_is_stale(updated_at + 50, 100) - .unwrap() - ); - assert!( - storage - .log_gas_price_is_stale(updated_at + 101, 100) - .unwrap() - ); } } diff --git a/sequencer/src/storage/history.rs b/sequencer/src/storage/history.rs new file mode 100644 index 00000000..cf2eb61d --- /dev/null +++ b/sequencer/src/storage/history.rs @@ -0,0 +1,376 @@ +// (c) Cartesi and individual authors (see AUTHORS) +// SPDX-License-Identifier: Apache-2.0 (see LICENSE) + +//! Current-era history metadata persisted as one SQLite singleton. + +use rusqlite::{Connection, Result, types::Type}; +use sequencer_core::history::{EraId, ExecutedInputCount, HistoryVersion, RecoveryGeneration}; + +use super::Storage; +use super::convert::{i64_to_u64, u64_to_i64}; + +/// Durable metadata for the history served by this database. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct HistoryState { + pub version: HistoryVersion, + pub era_created_at_ms: u64, + /// `None` only while an admitted cockroach rebuild has not yet registered + /// its recovered finalized application state. + pub base_executed_input_count: Option, + /// Exclusive `safe_inputs` cursor below which this era must never drain. + /// `None` has the same narrow pre-fill rebuild meaning as the application + /// base above; the two fields bind atomically. + pub base_safe_input_index: Option, +} + +/// One sparse attribution from SQLite's physical replay log to the canonical +/// application-history coordinate consumed by that row. +/// +/// Physical rows that do not execute in the application (our own submitted +/// batches and cockroach-root padding) deliberately have no mapping. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) struct ExecutedInputMapping { + pub sequenced_l2_tx_offset: u64, + pub executed_input_offset: ExecutedInputCount, +} + +/// One safe input from a drained physical range that actually executed in the +/// application. The range may also contain intentionally-unmapped rows, so the +/// caller supplies only these sparse attributions. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct DirectInputExecution { + pub safe_input_index: u64, + pub executed_input_offset: ExecutedInputCount, +} + +impl Storage { + /// Read the current era, recovery generation, and locally available base. + pub fn history_state(&self) -> Result { + query_history_state(&self.conn) + } + + /// Boundary before the next canonical application input executes. + /// + /// This is derived, never independently advanced: the maximum of the era's + /// recovered base and one past the greatest valid execution attribution. + /// Invalidating a suffix therefore rolls the value back automatically. + pub fn next_executed_input_count(&mut self) -> Result { + self.read(|tx| next_executed_input_count_in(tx)) + } +} + +pub(super) fn query_history_state(conn: &Connection) -> Result { + conn.query_row( + "SELECT era_id, era_created_at_ms, recovery_generation, \ + base_executed_input_count, base_safe_input_index \ + FROM history_state WHERE singleton_id = 0", + [], + |row| { + let era_blob = row.get::<_, Vec>(0)?; + let era_id = EraId::try_from(era_blob.as_slice()).map_err(|error| { + rusqlite::Error::FromSqlConversionFailure(0, Type::Blob, Box::new(error)) + })?; + Ok(HistoryState { + version: HistoryVersion { + era_id, + recovery_generation: RecoveryGeneration::new(i64_to_u64(row.get(2)?)), + }, + era_created_at_ms: i64_to_u64(row.get(1)?), + base_executed_input_count: row.get::<_, Option>(3)?.map(i64_to_u64), + base_safe_input_index: row.get::<_, Option>(4)?.map(i64_to_u64), + }) + }, + ) +} + +/// Derive the next canonical application coordinate from durable facts. +pub(super) fn next_executed_input_count_in(conn: &Connection) -> Result { + let base = query_history_state(conn)? + .base_executed_input_count + .expect("application history base is unbound outside rebuild fill"); + let greatest_valid: Option = conn.query_row( + "SELECT MAX(executed_input_offset) FROM executed_inputs", + [], + |row| row.get(0), + )?; + let after_valid = greatest_valid.map_or(0, |offset| { + i64_to_u64(offset) + .checked_add(1) + .expect("executed input offset overflow: contract-impossible") + }); + Ok(ExecutedInputCount::new(base.max(after_valid))) +} + +/// Attach a sequence of explicit application offsets inside the physical-row +/// creation transaction. Each offset must equal the currently-derived next +/// boundary; the baseline schema independently enforces the same rule over the +/// valid projection, including offset reuse after suffix invalidation. +pub(super) fn attach_executed_inputs_in( + tx: &rusqlite::Transaction<'_>, + mappings: &[ExecutedInputMapping], +) -> Result<()> { + if mappings.is_empty() { + return Ok(()); + } + + let mut expected = next_executed_input_count_in(tx)?; + let mut stmt = tx.prepare_cached( + "INSERT INTO executed_inputs \ + (sequenced_l2_tx_offset, executed_input_offset) VALUES (?1, ?2)", + )?; + for mapping in mappings { + assert_eq!( + mapping.executed_input_offset, expected, + "executed input attribution does not match canonical next count" + ); + stmt.execute(rusqlite::params![ + u64_to_i64(mapping.sequenced_l2_tx_offset), + u64_to_i64(mapping.executed_input_offset.get()), + ])?; + expected = expected + .checked_next() + .expect("executed input count overflow: contract-impossible"); + } + Ok(()) +} + +/// Bind the era's locally-available application base and durable safe-input +/// drain floor to the snapshot that establishes them. Genesis is already +/// initialized to zero by the baseline migration; rebuild starts with both +/// `NULL` and reaches this function with the folded application's absolute +/// count plus the recovery root's exclusive safe-input cursor. +pub(super) fn bind_history_base_in( + tx: &rusqlite::Transaction<'_>, + base_executed_input_count: u64, + base_safe_input_index: u64, +) -> Result<()> { + let current = query_history_state(tx)?; + match ( + current.base_executed_input_count, + current.base_safe_input_index, + ) { + (Some(current_count), Some(current_safe_input_index)) => { + assert_eq!( + current_count, base_executed_input_count, + "history base differs from the initial finalized application state" + ); + assert_eq!( + current_safe_input_index, base_safe_input_index, + "safe-input floor differs from the initial finalized application state" + ); + return Ok(()); + } + (None, None) => {} + _ => unreachable!("history base pair cannot be partially bound"), + } + + let changed = tx.execute( + "UPDATE history_state \ + SET base_executed_input_count = ?1, base_safe_input_index = ?2 \ + WHERE singleton_id = 0 \ + AND base_executed_input_count IS NULL \ + AND base_safe_input_index IS NULL", + rusqlite::params![ + u64_to_i64(base_executed_input_count), + u64_to_i64(base_safe_input_index) + ], + )?; + if changed != 1 { + return Err(rusqlite::Error::StatementChangedRows(changed)); + } + Ok(()) +} + +/// Durable lower bound for safe-input draining. A NULL floor is usable as zero +/// only while a rebuild's setup has not completed: that is the one interval +/// in which the recovery root must be populated before its cursor can be +/// bound. Everywhere else NULL is a storage invariant violation and fails +/// loud. +pub(super) fn safe_input_floor_in(conn: &Connection) -> Result { + let state = query_history_state(conn)?; + match state.base_safe_input_index { + Some(floor) => Ok(floor), + None => { + assert!( + state.base_executed_input_count.is_none(), + "history base pair cannot be partially bound" + ); + // A NULL floor exists only during a pre-completion rebuild + // fill: plain setup binds base 0 in its baseline transaction, and + // completion refuses while the base is NULL — so the completion + // fact alone decides legality (the journal is never read for + // decisions; L2). + let setup_complete: bool = conn.query_row( + "SELECT EXISTS (SELECT 1 FROM setup_complete WHERE singleton_id = 0)", + [], + |row| row.get(0), + )?; + assert!( + !setup_complete, + "NULL safe-input floor outside pre-completion rebuild fill" + ); + Ok(0) + } + } +} + +/// Advance the current era's soft-history reality by exactly one. The schema +/// independently rejects skips and rewrites; the caller composes this helper +/// into the same transaction as suffix invalidation and Tip reopening. +pub(super) fn advance_recovery_generation_in( + tx: &rusqlite::Transaction<'_>, +) -> Result { + let current: i64 = tx.query_row( + "SELECT recovery_generation FROM history_state WHERE singleton_id = 0", + [], + |row| row.get(0), + )?; + let next = current + .checked_add(1) + .expect("recovery generation exhausted SQLite INTEGER: contract-impossible"); + let changed = tx.execute( + "UPDATE history_state SET recovery_generation = ?1 WHERE singleton_id = 0", + [next], + )?; + if changed != 1 { + return Err(rusqlite::Error::StatementChangedRows(changed)); + } + Ok(RecoveryGeneration::new(i64_to_u64(next))) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::storage::test_helpers::temp_db; + use crate::storage::{LifecycleCommand, Storage}; + + #[test] + fn history_schema_enforces_write_once_identity_and_base() { + let db = temp_db("history-write-once"); + let storage = Storage::open(db.path.as_str()).expect("initialize generic history"); + let original = storage.history_state().expect("read history"); + drop(storage); + + let conn = Storage::open_connection(db.path.as_str()).expect("open raw connection"); + assert!( + conn.execute( + "UPDATE history_state SET era_id = era_id WHERE singleton_id = 0", + [], + ) + .is_err(), + "era identity must reject even a same-value rewrite" + ); + assert!( + conn.execute( + "UPDATE history_state SET base_executed_input_count = 1 \ + WHERE singleton_id = 0", + [], + ) + .is_err(), + "the initialized genesis base must be immutable" + ); + assert!( + conn.execute( + "UPDATE history_state SET base_safe_input_index = 1 \ + WHERE singleton_id = 0", + [], + ) + .is_err(), + "the initialized genesis safe-input floor must be immutable" + ); + assert!( + conn.execute("DELETE FROM history_state WHERE singleton_id = 0", []) + .is_err(), + "the current era singleton must not be deletable" + ); + + let reopened = Storage::open_read_only(db.path.as_str()).expect("reopen"); + assert_eq!(reopened.history_state().expect("read history"), original); + } + + #[test] + fn rebuild_base_is_set_once_and_generation_advances_only_by_one() { + let db = temp_db("history-rebuild-transitions"); + let storage = Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Rebuild) + .expect("initialize rebuild"); + assert_eq!( + storage + .history_state() + .expect("read pending rebuild") + .base_executed_input_count, + None + ); + assert_eq!( + storage + .history_state() + .expect("read pending rebuild") + .base_safe_input_index, + None + ); + drop(storage); + + let conn = Storage::open_connection(db.path.as_str()).expect("open raw connection"); + assert!( + conn.execute( + "UPDATE history_state SET base_executed_input_count = 41 \ + WHERE singleton_id = 0", + [], + ) + .is_err(), + "the application base cannot bind without its safe-input floor" + ); + conn.execute( + "UPDATE history_state \ + SET base_executed_input_count = 41, base_safe_input_index = 7 \ + WHERE singleton_id = 0", + [], + ) + .expect("set rebuild base pair once"); + assert!( + conn.execute( + "UPDATE history_state SET base_executed_input_count = 42 \ + WHERE singleton_id = 0", + [], + ) + .is_err(), + "rebuild base must not be rewritten" + ); + assert!( + conn.execute( + "UPDATE history_state SET base_safe_input_index = 8 \ + WHERE singleton_id = 0", + [], + ) + .is_err(), + "rebuild safe-input floor must not be rewritten" + ); + + conn.execute( + "UPDATE history_state SET recovery_generation = 1 WHERE singleton_id = 0", + [], + ) + .expect("advance generation by one"); + assert!( + conn.execute( + "UPDATE history_state SET recovery_generation = 3 WHERE singleton_id = 0", + [], + ) + .is_err(), + "generation must not skip" + ); + conn.execute( + "UPDATE history_state SET recovery_generation = 2 WHERE singleton_id = 0", + [], + ) + .expect("advance generation by one again"); + + let reopened = Storage::open_read_only(db.path.as_str()).expect("reopen"); + let state = reopened.history_state().expect("read history"); + assert_eq!(state.base_executed_input_count, Some(41)); + assert_eq!(state.base_safe_input_index, Some(7)); + assert_eq!( + state.version.recovery_generation, + RecoveryGeneration::new(2) + ); + } +} diff --git a/sequencer/src/storage/ingress.rs b/sequencer/src/storage/ingress.rs index 1d379306..def35265 100644 --- a/sequencer/src/storage/ingress.rs +++ b/sequencer/src/storage/ingress.rs @@ -13,26 +13,39 @@ use std::path::Path; use alloy_primitives::Address; use rusqlite::{Result, Transaction, params}; -use super::convert::{from_unix_ms, i64_to_u64, now_unix_ms, to_unix_ms, u64_to_i64}; +use super::convert::{ + external_u64_to_i64, from_unix_ms, i64_to_u64, now_unix_ms, saturating_query_bound, to_unix_ms, + u64_to_i64, +}; +use super::history::{ExecutedInputMapping, attach_executed_inputs_in, safe_input_floor_in}; use super::mutations::{ - insert_new_batch, insert_open_frame, persist_frame_direct_sequence, seal_batch, + insert_new_batch, insert_open_frame, persist_frame_direct_sequence, + persist_frame_direct_sequence_derived, persist_frame_direct_sequence_physical_only, seal_batch, }; use super::queries::{ current_safe_block_required, load_current_write_head, query_batch_policy, query_latest_safe_input_index_exclusive, valid_ordered_l2_tx_head, }; +use super::safe_accepted_batches::canonical_divergence_in; use super::snapshot_dumps::{insert_pending_dump_in, promote_finalized_in}; -use super::{BatchPolicy, SafeInputFrontier, SafeInputRange, Storage, StoredSafeInput, WriteHead}; -use crate::ingress::inclusion_lane::PendingUserOp; +use super::{ + BatchPolicy, DirectInputExecution, ExecutedInputCount, SafeFrontierState, SafeInputFrontier, + SafeInputRange, Storage, StoredSafeInput, WriteHead, +}; +use crate::ingress::inclusion_lane::{IncludedUserOp, PendingUserOp}; impl Storage { - /// Cursor for the next safe input to drain into a frame. Reads the highest - /// already-drained `safe_input_index` from the valid (non-invalidated) - /// `sequenced_l2_txs` rows and returns `MAX + 1` (or 0 if none). + /// Cursor for the next safe input to drain into a frame. Takes the maximum + /// of the era's durable drain floor and the highest already-drained + /// `safe_input_index` from valid (non-invalidated) `sequenced_l2_txs` rows + /// plus one. /// /// Using `MAX + 1` instead of `COUNT(*)` makes this robust against gaps: /// when a batch is invalidated, those rows drop out of the view and the - /// cursor naturally rewinds, allowing the recovery batch to re-drain. + /// cursor naturally rewinds, allowing the recovery batch to re-drain only + /// the invalidated suffix. The durable floor prevents a standard recovery + /// from crossing back into cockroach-root padding already represented by + /// the recovered application snapshot. pub fn next_undrained_safe_input_index(&mut self) -> Result { self.read(next_undrained_safe_input_index_in) } @@ -46,10 +59,11 @@ impl Storage { /// Bootstrap the very first batch + frame with explicit values, returning /// its loaded [`WriteHead`]. Asserts no open state exists. /// - /// Production opens the genesis Tip through [`Storage::ensure_open_tip`], - /// which derives `safe_block`/leading range from the synced L1 view; this - /// explicit form is kept for tests that seed a specific open state without - /// a safe-head observation. + /// Production opens the genesis Tip through the reducer's guarded + /// `EnsureOpenTip` phase, which derives `safe_block`/leading range from the + /// synced L1 view; this explicit form is kept for tests that seed a + /// specific open state without a safe-head observation. + #[cfg(test)] pub fn initialize_open_state( &mut self, safe_block: u64, @@ -60,7 +74,8 @@ impl Storage { load_current_write_head(tx)?.is_none(), "open state already exists" ); - insert_tip_rows(tx, Some(0), None, safe_block, leading_direct_range)?; + let batch_index = insert_tip_rows(tx, Some(0), None, safe_block)?; + persist_frame_direct_sequence_physical_only(tx, batch_index, 0, leading_direct_range)?; Ok(load_current_write_head(tx)?.expect("genesis tip just inserted")) }) } @@ -77,12 +92,10 @@ impl Storage { /// Those directs are executed by the lane's catch-up replay (the same path /// warm resume and recovery batches use), so there is no cold-start drain. /// - /// This is the genesis / first-startup guard. The runtime calls it once at - /// startup — after recovery has synced the safe head and the genesis - /// snapshot is registered, immediately before the lane starts — so the lane - /// only ever *loads* a Tip (fail-loud if absent) and never initializes one. - /// Recovery owns its own atomic reopen (it repairs a tip-lessness it - /// created inside the cascade transaction), reusing the same mechanism. + /// This unguarded public form exists for test harnesses. Production uses + /// `ensure_open_tip_for_recovery`, which reasserts the reducer facts in its + /// write transaction. The lane only ever *loads* a Tip (fail-loud if + /// absent); Cascade owns its own atomic reopen using the same mechanism. /// /// **Precondition (genesis branch only):** a safe-head observation must /// exist. A fresh DB requires L1 at bootstrap plus a successful recovery @@ -113,22 +126,29 @@ impl Storage { /// while the scheduler drains them on-chain (divergence). Capping the drain /// at `C` leaves `(C, H1]` undrained, so `run`'s lane leads and executes them /// exactly once as the safe frontier advances `C -> H1`. - pub fn open_recovery_tip(&mut self, stop_block: u64) -> Result<()> { + pub(crate) fn open_recovery_tip(&mut self, stop_block: u64) -> Result<()> { + external_u64_to_i64(stop_block, "recovery checkpoint block")?; self.write(|tx| open_recovery_tip_in_tx(tx, stop_block)) } - /// Snapshot the current L1 view: safe block + exclusive safe-input cursor. - /// The lane uses this to decide whether to advance. + /// Snapshot the current L1 reconciliation state. A canonical-divergence + /// marker outranks and withholds the otherwise-usable frontier. /// /// **Precondition:** at least one safe-head observation must have been - /// recorded. The lane only starts after `run_preemptive_recovery` - /// completes, which guarantees this in production. - pub fn safe_input_frontier(&mut self) -> Result { + /// recorded. The lane only starts after the recovery reducer admits, + /// which guarantees this in production. + pub fn safe_frontier_state(&mut self) -> Result { self.read(|tx| { - Ok(SafeInputFrontier { + if let Some((nonce, safe_input_index)) = canonical_divergence_in(tx)? { + return Ok(SafeFrontierState::CanonicalDivergence { + nonce, + safe_input_index, + }); + } + Ok(SafeFrontierState::Open(SafeInputFrontier { safe_block: current_safe_block_required(tx)?, end_exclusive: query_latest_safe_input_index_exclusive(tx)?, - }) + })) }) } @@ -168,7 +188,12 @@ impl Storage { for (offset, row) in rows.enumerate() { let (index_i64, sender, payload, block_number_i64) = row?; let index = i64_to_u64(index_i64); - let expected = range.start().saturating_add(offset as u64); + let offset = u64::try_from(offset) + .expect("safe-input result offset exceeds u64: contract-impossible"); + let expected = range + .start() + .checked_add(offset) + .expect("safe-input expected index overflow: contract-impossible"); assert_eq!( index, expected, @@ -180,11 +205,17 @@ impl Storage { payload, block_number: i64_to_u64(block_number_i64), }); - fetched_count = fetched_count.saturating_add(1); + fetched_count = fetched_count + .checked_add(1) + .expect("safe-input fetched count overflow: contract-impossible"); } + let fetched_end = range + .start() + .checked_add(fetched_count) + .expect("safe-input fetched range overflow: contract-impossible"); assert_eq!( - range.start().saturating_add(fetched_count), + fetched_end, range.end(), "safe-input range {range:?} not fully populated" ); @@ -195,11 +226,42 @@ impl Storage { /// Persist a chunk of user ops into the open frame and bump `head`'s /// counters. /// - /// `head` is treated as authoritative: the lane is the only writer of - /// open-frame state, so a stale `WriteHead` indicates a bug in the lane, - /// not a runtime condition. The schema's FK + PK constraints catch the - /// dangerous failure modes (write to a non-existent frame, duplicate - /// `pos_in_frame`) by failing the INSERT. + /// `head` is trusted as a coherent cache: SQLite is durable authority, and + /// the lane is the only writer of open-frame state. A stale `WriteHead` + /// therefore indicates a bug in the lane, not a runtime condition. The + /// schema's FK + PK constraints catch the dangerous failure modes (write + /// to a non-existent frame, duplicate `pos_in_frame`) by failing the + /// INSERT. + pub(crate) fn append_executed_user_ops_chunk( + &mut self, + head: &mut WriteHead, + user_ops: &[IncludedUserOp], + ) -> Result<()> { + if user_ops.is_empty() { + return Ok(()); + } + // Validate both in-memory counter advances before the transaction. + // Otherwise an overflow panic after commit would leave durable rows + // that the unchanged `WriteHead` cannot describe. + let mut next_head = *head; + next_head.increment_batch_user_op_count(user_ops.len()); + self.write(|tx| { + insert_executed_user_ops_batch( + tx, + head.batch_index, + head.frame_in_batch, + head.open_frame_user_op_count, + user_ops, + ) + })?; + *head = next_head; + Ok(()) + } + + /// Physical-only fixture writer. Production must use + /// [`Storage::append_executed_user_ops_chunk`] so creation and canonical + /// execution attribution commit atomically. + #[cfg(test)] pub fn append_user_ops_chunk( &mut self, head: &mut WriteHead, @@ -208,6 +270,8 @@ impl Storage { if user_ops.is_empty() { return Ok(()); } + let mut next_head = *head; + next_head.increment_batch_user_op_count(user_ops.len()); self.write(|tx| { insert_user_ops_batch( tx, @@ -217,21 +281,40 @@ impl Storage { user_ops, ) })?; - head.increment_batch_user_op_count(user_ops.len()); + *head = next_head; Ok(()) } /// Rotate to the next frame inside the same batch. Used when the safe /// block advances but batch policy hasn't triggered a batch close — the /// new frame inherits the batch and gets a fresh fee/safe-block. + pub fn close_frame_only_with_executions( + &mut self, + head: &mut WriteHead, + next_safe_block: u64, + leading_direct_range: SafeInputRange, + executions: &[DirectInputExecution], + ) -> Result<()> { + let policy = self.write(|tx| { + close_frame_in(tx, head, next_safe_block, leading_direct_range, executions) + })?; + head.advance_frame(policy, next_safe_block); + Ok(()) + } + + /// Physical-only fixture frame rotation. Production must supply explicit + /// execution attributions through + /// [`Storage::close_frame_only_with_executions`]. + #[cfg(test)] pub fn close_frame_only( &mut self, head: &mut WriteHead, next_safe_block: u64, leading_direct_range: SafeInputRange, ) -> Result<()> { - let policy = - self.write(|tx| close_frame_in(tx, head, next_safe_block, leading_direct_range))?; + let policy = self.write(|tx| { + close_frame_physical_only_in(tx, head, next_safe_block, leading_direct_range) + })?; head.advance_frame(policy, next_safe_block); Ok(()) } @@ -246,6 +329,27 @@ impl Storage { /// the state where a restart re-processes the safe input, re-derives the /// accepted nonce, and re-promotes on a now-deleted pending row /// (`QueryReturnedNoRows`, a fail-loud wedge). + pub fn close_frame_only_promoting_with_executions( + &mut self, + head: &mut WriteHead, + next_safe_block: u64, + leading_direct_range: SafeInputRange, + executions: &[DirectInputExecution], + max_nonce: u64, + inclusion_block: u64, + ) -> Result<()> { + let policy = self.write(|tx| { + let policy = + close_frame_in(tx, head, next_safe_block, leading_direct_range, executions)?; + promote_finalized_in(tx, max_nonce, inclusion_block)?; + Ok(policy) + })?; + head.advance_frame(policy, next_safe_block); + Ok(()) + } + + /// Physical-only fixture form of the atomic drain + promotion operation. + #[cfg(test)] pub fn close_frame_only_promoting( &mut self, head: &mut WriteHead, @@ -255,7 +359,8 @@ impl Storage { inclusion_block: u64, ) -> Result<()> { let policy = self.write(|tx| { - let policy = close_frame_in(tx, head, next_safe_block, leading_direct_range)?; + let policy = + close_frame_physical_only_in(tx, head, next_safe_block, leading_direct_range)?; promote_finalized_in(tx, max_nonce, inclusion_block)?; Ok(policy) })?; @@ -312,6 +417,7 @@ impl Storage { dump_dir: &Path, nonce: u64, l2_tx_index: u64, + executed_input_count: ExecutedInputCount, ) -> Result<()> { let (next_batch_index, now_ms, policy) = self.write(|tx| { // The lane is the single writer and nothing sequences between @@ -325,7 +431,7 @@ impl Storage { ); let (next_batch_index, now_ms, policy) = seal_and_open_next_batch(tx, head.batch_index, next_safe_block)?; - insert_pending_dump_in(tx, dump_dir, nonce, l2_tx_index)?; + insert_pending_dump_in(tx, dump_dir, nonce, l2_tx_index, executed_input_count)?; Ok((next_batch_index, now_ms, policy)) })?; head.move_to_next_batch( @@ -342,10 +448,10 @@ impl Storage { } } -/// Insert a fresh open batch (the Tip) and its first frame inside `tx`, -/// sequencing `leading_direct_range` into that frame (**sequenced, not -/// executed** — the app catches up by replaying the same rows). Lineage is the -/// caller's: `batch_index_opt = Some(0)` forces the genesis index, `None` +/// Insert a fresh open batch (the Tip) and its first empty frame inside `tx`. +/// The caller immediately persists the leading direct range through either the +/// production attributed path or the explicit cockroach/test padding path. +/// Lineage is the caller's: `batch_index_opt = Some(0)` forces the genesis index, `None` /// auto-assigns the PK; `parent = None` roots a nonce-0 batch (genesis or a /// fully-torn refork), else it inherits `parent.nonce + 1`. The single-Tip /// invariant is enforced by the `ux_single_valid_tip` partial index. @@ -358,7 +464,6 @@ fn insert_tip_rows( batch_index_opt: Option, parent: Option, safe_block: u64, - leading_direct_range: SafeInputRange, ) -> Result { let now_ms = now_unix_ms(); let policy = query_batch_policy(tx)?; @@ -371,7 +476,6 @@ fn insert_tip_rows( policy.recommended_fee, safe_block, )?; - persist_frame_direct_sequence(tx, batch_index, 0, leading_direct_range)?; Ok(batch_index) } @@ -400,7 +504,7 @@ pub(super) fn open_fresh_tip_in_tx(tx: &Transaction<'_>) -> Result<()> { })? .map(i64_to_u64); let batch_index_opt = if table_empty { Some(0) } else { None }; - insert_draining_tip( + insert_draining_tip_with_executions( tx, batch_index_opt, parent, @@ -416,7 +520,7 @@ pub(super) fn open_fresh_tip_in_tx(tx: &Transaction<'_>) -> Result<()> { /// on the recovery path (the load-bearing `(C, H1]` difference) — plus /// `safe_block` and lineage, which they pass explicitly so the cap rule stays /// visible at each call site rather than hidden in one branch. -fn insert_draining_tip( +fn insert_draining_tip_with_executions( tx: &Transaction<'_>, batch_index: Option, parent: Option, @@ -425,7 +529,8 @@ fn insert_draining_tip( ) -> Result<()> { let leading_direct_range = SafeInputRange::new(next_undrained_safe_input_index_in(tx)?, drain_upper); - insert_tip_rows(tx, batch_index, parent, safe_block, leading_direct_range)?; + let batch_index = insert_tip_rows(tx, batch_index, parent, safe_block)?; + persist_frame_direct_sequence_derived(tx, batch_index, 0, leading_direct_range)?; Ok(()) } @@ -450,13 +555,15 @@ fn open_recovery_tip_in_tx(tx: &Transaction<'_>, stop_block: u64) -> Result<()> // The `sender != batch_submitter` drop belongs to the *fold's* seed filter // (those batches were folded into `S'` as batches, not directs); it is not a // cursor concern, so it deliberately does not reappear here. - insert_draining_tip( - tx, - Some(0), - None, - stop_block, + let leading_direct_range = SafeInputRange::new( + next_undrained_safe_input_index_in(tx)?, safe_input_index_exclusive_through_block_in(tx, stop_block)?, - ) + ); + // These rows are cursor padding for state already represented by the + // recovered snapshot. They permanently remain outside `executed_inputs`. + let batch_index = insert_tip_rows(tx, Some(0), None, stop_block)?; + persist_frame_direct_sequence_physical_only(tx, batch_index, 0, leading_direct_range)?; + Ok(()) } /// Exclusive `safe_input_index` boundary separating directs at `block_number <= @@ -466,15 +573,16 @@ fn open_recovery_tip_in_tx(tx: &Transaction<'_>, stop_block: u64) -> Result<()> fn safe_input_index_exclusive_through_block_in(tx: &Transaction<'_>, block: u64) -> Result { let boundary: i64 = tx.query_row( "SELECT COUNT(*) FROM safe_inputs WHERE block_number <= ?1", - params![u64_to_i64(block)], + params![saturating_query_bound(block)], |row| row.get(0), )?; Ok(i64_to_u64(boundary)) } -/// `MAX(safe_input_index) + 1` over the valid drained rows (or 0 if none), -/// inside `tx`. The cursor rewinds when a batch is invalidated, so a recovery -/// batch re-drains the same range its invalidated predecessor was working from. +/// Maximum of the durable era floor and `MAX(safe_input_index) + 1` over valid +/// drained rows (or 0 if none), inside `tx`. The valid attribution may rewind +/// when a batch is invalidated, but never below inputs already represented by +/// the cockroach-recovered base snapshot. fn next_undrained_safe_input_index_in(tx: &Transaction<'_>) -> Result { const SQL: &str = " SELECT COALESCE(MAX(safe_input_index) + 1, 0) @@ -482,7 +590,7 @@ fn next_undrained_safe_input_index_in(tx: &Transaction<'_>) -> Result { WHERE safe_input_index IS NOT NULL "; let value: i64 = tx.query_row(SQL, [], |row| row.get(0))?; - Ok(i64_to_u64(value)) + Ok(safe_input_floor_in(tx)?.max(i64_to_u64(value))) } /// Seal the current Tip and open the successor batch's first frame, in `tx`. @@ -502,7 +610,7 @@ fn seal_and_open_next_batch( // Batch policy is sampled here: the derived fee is committed to the newly // opened frame, and the batch size target is stored on the write head. let policy = query_batch_policy(tx)?; - // Hash-at-seal (review R2): encode the closing batch's wire bytes via + // Hash-at-seal: encode the closing batch's wire bytes via // the same path the submitter uses and stamp their keccak256 on the row, // atomically with the seal. The content-identity check later compares // accepted L1 landings against this hash. @@ -525,28 +633,37 @@ fn seal_and_open_next_batch( /// Rotate to the next frame inside the current batch, in `tx`: open the /// successor frame (fresh fee/safe-block) and sequence the drained safe-input -/// range into it. Shared by [`Storage::close_frame_only`] and -/// [`Storage::close_frame_only_promoting`] so the frame-rotation invariant -/// lives in one place. Returns the sampled policy for the caller to apply to -/// the in-memory write head. +/// range into it. Shared by the attributed production close/promote paths and +/// their physical-only test siblings so the frame-rotation invariant lives in +/// one place. Returns the sampled policy for the caller to apply to the +/// in-memory write head. fn close_frame_in( tx: &Transaction<'_>, head: &WriteHead, next_safe_block: u64, leading_direct_range: SafeInputRange, + executions: &[DirectInputExecution], ) -> Result { - let now_ms = now_unix_ms(); - let policy = query_batch_policy(tx)?; - let next_frame_in_batch = head.frame_in_batch.saturating_add(1); - insert_open_frame( + let (policy, next_frame_in_batch) = open_successor_frame_in(tx, head, next_safe_block)?; + persist_frame_direct_sequence( tx, head.batch_index, next_frame_in_batch, - now_ms, - policy.recommended_fee, - next_safe_block, + leading_direct_range, + executions, )?; - persist_frame_direct_sequence( + Ok(policy) +} + +#[cfg(test)] +fn close_frame_physical_only_in( + tx: &Transaction<'_>, + head: &WriteHead, + next_safe_block: u64, + leading_direct_range: SafeInputRange, +) -> Result { + let (policy, next_frame_in_batch) = open_successor_frame_in(tx, head, next_safe_block)?; + persist_frame_direct_sequence_physical_only( tx, head.batch_index, next_frame_in_batch, @@ -555,8 +672,31 @@ fn close_frame_in( Ok(policy) } +fn open_successor_frame_in( + tx: &Transaction<'_>, + head: &WriteHead, + next_safe_block: u64, +) -> Result<(BatchPolicy, u32)> { + let now_ms = now_unix_ms(); + let policy = query_batch_policy(tx)?; + let next_frame_in_batch = head + .frame_in_batch + .checked_add(1) + .expect("frame index overflow: contract-impossible"); + insert_open_frame( + tx, + head.batch_index, + next_frame_in_batch, + now_ms, + policy.recommended_fee, + next_safe_block, + )?; + Ok((policy, next_frame_in_batch)) +} + /// Insert user ops into `user_ops`. The `trg_sequence_user_op` trigger then /// appends the matching `sequenced_l2_txs` row for each insert. +#[cfg(test)] fn insert_user_ops_batch( tx: &Transaction<'_>, batch_index: u64, @@ -564,17 +704,34 @@ fn insert_user_ops_batch( frame_pos_start: u32, user_ops: &[PendingUserOp], ) -> Result<()> { - if user_ops.is_empty() { - return Ok(()); - } + insert_user_op_iter( + tx, + batch_index, + frame_in_batch, + frame_pos_start, + user_ops.iter(), + ) +} + +fn insert_user_op_iter<'a>( + tx: &Transaction<'_>, + batch_index: u64, + frame_in_batch: u32, + frame_pos_start: u32, + user_ops: impl IntoIterator, +) -> Result<()> { let mut stmt = tx.prepare_cached( "INSERT INTO user_ops ( batch_index, frame_in_batch, pos_in_frame, sender, nonce, max_fee, data, sig, received_at_ms ) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)", )?; - for (offset, item) in user_ops.iter().enumerate() { - let pos_in_frame = frame_pos_start.saturating_add(offset as u32); + for (offset, item) in user_ops.into_iter().enumerate() { + let offset = + u32::try_from(offset).expect("user-op chunk offset exceeds u32: contract-impossible"); + let pos_in_frame = frame_pos_start + .checked_add(offset) + .expect("user-op position overflow: contract-impossible"); let sig = item.signed.signature.as_bytes(); stmt.execute(params![ u64_to_i64(batch_index), @@ -591,14 +748,150 @@ fn insert_user_ops_batch( Ok(()) } +/// Persist one included user-op chunk and attach every explicit application +/// offset before the transaction can commit. The trigger-created physical rows +/// are selected in frame-position order once per chunk, keeping the hot path to +/// one attribution read rather than one lookup per operation. +fn insert_executed_user_ops_batch( + tx: &Transaction<'_>, + batch_index: u64, + frame_in_batch: u32, + frame_pos_start: u32, + user_ops: &[IncludedUserOp], +) -> Result<()> { + insert_user_op_iter( + tx, + batch_index, + frame_in_batch, + frame_pos_start, + user_ops.iter().map(|item| &item.pending), + )?; + if user_ops.is_empty() { + return Ok(()); + } + + let chunk_len = u32::try_from(user_ops.len()) + .expect("user-op chunk length exceeds u32: contract-impossible"); + let frame_pos_end = frame_pos_start + .checked_add(chunk_len) + .expect("user-op position overflow: contract-impossible"); + let mut stmt = tx.prepare_cached( + "SELECT offset, user_op_pos_in_frame \ + FROM sequenced_l2_txs \ + WHERE batch_index = ?1 \ + AND frame_in_batch = ?2 \ + AND user_op_pos_in_frame >= ?3 \ + AND user_op_pos_in_frame < ?4 \ + ORDER BY user_op_pos_in_frame ASC", + )?; + let rows = stmt.query_map( + params![ + u64_to_i64(batch_index), + i64::from(frame_in_batch), + i64::from(frame_pos_start), + i64::from(frame_pos_end), + ], + |row| Ok((row.get::<_, i64>(0)?, row.get::<_, i64>(1)?)), + )?; + let persisted = rows.collect::>>()?; + drop(stmt); + assert_eq!( + persisted.len(), + user_ops.len(), + "user-op physical row count differs from included execution count" + ); + + let mut mappings = Vec::with_capacity(persisted.len()); + for (offset, ((physical_offset, position), executed_input_offset)) in persisted + .into_iter() + .zip(user_ops.iter().map(|item| item.executed_input_offset)) + .enumerate() + { + let offset = + u32::try_from(offset).expect("user-op chunk offset exceeds u32: contract-impossible"); + assert_eq!( + i64_to_u64(position), + u64::from( + frame_pos_start + .checked_add(offset) + .expect("user-op position overflow: contract-impossible") + ), + "user-op physical rows are not contiguous in frame order" + ); + mappings.push(ExecutedInputMapping { + sequenced_l2_tx_offset: i64_to_u64(physical_offset), + executed_input_offset, + }); + } + attach_executed_inputs_in(tx, &mappings) +} + #[cfg(test)] mod tests { + use crate::ingress::inclusion_lane::{IncludedUserOp, PendingUserOp}; use crate::storage::{ - SafeInputRange, Storage, StoredSafeInput, - test_helpers::{SENDER_A, default_protocol_timing, temp_db}, + DeploymentIdentity, DirectInputExecution, ExecutedInputCount, FeeOracleIdentity, + LifecycleCommand, SafeFrontierState, SafeInputFrontier, SafeInputRange, Storage, + StoredSafeInput, + test_helpers::{SENDER_A, default_protocol_timing, record_canonical_divergence, temp_db}, }; - use alloy_primitives::Address; + use alloy_primitives::{Address, Signature}; use sequencer_core::l2_tx::SequencedL2Tx; + use sequencer_core::user_op::{SignedUserOp, UserOp}; + use std::panic::{AssertUnwindSafe, catch_unwind}; + use std::time::SystemTime; + use tokio::sync::oneshot; + + fn pending_user_op(nonce: u32) -> PendingUserOp { + let (respond_to, _response) = oneshot::channel(); + PendingUserOp { + signed: SignedUserOp { + sender: Address::ZERO, + signature: Signature::test_signature(), + user_op: UserOp { + nonce, + max_fee: u16::MAX, + data: vec![].into(), + }, + }, + respond_to, + received_at: SystemTime::now(), + } + } + + fn included_user_op(nonce: u32, offset: u64) -> IncludedUserOp { + IncludedUserOp { + pending: pending_user_op(nonce), + executed_input_offset: ExecutedInputCount::new(offset), + } + } + + fn physical_user_op_offsets(storage: &Storage) -> Vec { + storage + .conn + .prepare( + "SELECT offset FROM sequenced_l2_txs \ + WHERE user_op_pos_in_frame IS NOT NULL ORDER BY offset", + ) + .expect("prepare physical user-op query") + .query_map([], |row| row.get(0)) + .expect("query physical user-op offsets") + .collect::>>() + .expect("collect physical user-op offsets") + } + + fn pin_deployment_identity(storage: &mut Storage, batch_submitter_address: Address) { + storage + .load_or_insert_deployment_identity(DeploymentIdentity { + chain_id: 1, + app_address: Address::repeat_byte(0x11), + input_box_address: Address::repeat_byte(0x22), + app_deployment_block: 0, + batch_submitter_address, + fee_oracle: FeeOracleIdentity::Fixed { log_gas_price: 0 }, + }) + .expect("pin deployment identity"); + } #[test] fn open_state_is_idempotent_and_rotation_is_atomic() { @@ -641,6 +934,518 @@ mod tests { assert_eq!(head_d.frame_in_batch, 0); } + #[test] + fn safe_frontier_state_withholds_poisoned_projection() { + let db = temp_db("safe-frontier-state-poison"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + storage + .append_safe_inputs(12, &[], SENDER_A, &default_protocol_timing()) + .expect("record safe head"); + + assert_eq!( + storage.safe_frontier_state().expect("read open frontier"), + SafeFrontierState::Open(SafeInputFrontier { + safe_block: 12, + end_exclusive: 0, + }) + ); + + record_canonical_divergence(&mut storage, 7, 3); + assert_eq!( + storage + .safe_frontier_state() + .expect("read poisoned frontier"), + SafeFrontierState::CanonicalDivergence { + nonce: 7, + safe_input_index: 3, + }, + "the marker must outrank an otherwise-valid safe frontier" + ); + } + + #[test] + fn append_counter_overflow_happens_before_the_transaction() { + let db = temp_db("append-counter-overflow-before-write"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let mut head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + head.open_frame_user_op_count = u32::MAX; + + let (respond_to, _response) = oneshot::channel(); + let pending = PendingUserOp { + signed: SignedUserOp { + sender: Address::ZERO, + signature: Signature::test_signature(), + user_op: UserOp { + nonce: 0, + max_fee: u16::MAX, + data: vec![].into(), + }, + }, + respond_to, + received_at: SystemTime::now(), + }; + + let panic = catch_unwind(AssertUnwindSafe(|| { + let _ = storage.append_user_ops_chunk(&mut head, &[pending]); + })); + assert!(panic.is_err(), "counter overflow must fail loud"); + + let persisted: i64 = storage + .conn + .query_row("SELECT COUNT(*) FROM user_ops", [], |row| row.get(0)) + .expect("count user ops"); + assert_eq!(persisted, 0, "overflow must occur before any durable write"); + assert_eq!( + head.open_frame_user_op_count, + u32::MAX, + "the authoritative head must remain unchanged" + ); + } + + #[test] + fn mismatched_user_execution_offset_rolls_back_physical_and_logical_rows() { + let db = temp_db("user-execution-offset-atomicity"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let mut head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + + let included = included_user_op(0, 1); + let panic = catch_unwind(AssertUnwindSafe(|| { + let _ = storage.append_executed_user_ops_chunk(&mut head, &[included]); + })); + assert!( + panic.is_err(), + "non-canonical execution offset must fail loud" + ); + + for table in ["user_ops", "sequenced_l2_txs", "executed_inputs"] { + let persisted: i64 = storage + .conn + .query_row(&format!("SELECT COUNT(*) FROM {table}"), [], |row| { + row.get(0) + }) + .expect("count rolled-back rows"); + assert_eq!(persisted, 0, "{table} must roll back atomically"); + } + assert_eq!( + storage.next_executed_input_count().expect("next count"), + ExecutedInputCount::ZERO + ); + } + + #[test] + fn schema_rejects_execution_attribution_to_a_non_tip_row() { + let db = temp_db("executed-input-non-tip"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let mut head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + storage + .append_user_ops_chunk(&mut head, &[pending_user_op(0)]) + .expect("append physical-only user op"); + let physical_offset = physical_user_op_offsets(&storage)[0]; + storage + .close_frame_and_batch(&mut head, 0) + .expect("seal the physical row's batch"); + + let err = storage + .conn + .execute( + "INSERT INTO executed_inputs \ + (sequenced_l2_tx_offset, executed_input_offset) VALUES (?1, 0)", + [physical_offset], + ) + .expect_err("a sealed batch cannot acquire execution attribution"); + assert!( + err.to_string().contains("current valid Tip"), + "unexpected trigger error: {err:?}" + ); + } + + #[test] + fn schema_rejects_execution_attribution_before_rebuild_base_is_bound() { + let db = temp_db("executed-input-unbound-base"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Rebuild) + .expect("initialize rebuild storage"); + let mut head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize recovery root fixture"); + storage + .append_user_ops_chunk(&mut head, &[pending_user_op(0)]) + .expect("append physical-only user op"); + let physical_offset = physical_user_op_offsets(&storage)[0]; + + let err = storage + .conn + .execute( + "INSERT INTO executed_inputs \ + (sequenced_l2_tx_offset, executed_input_offset) VALUES (?1, 0)", + [physical_offset], + ) + .expect_err("an unbound rebuild cannot acquire execution attribution"); + assert!( + err.to_string().contains("history base is not bound"), + "unexpected trigger error: {err:?}" + ); + } + + #[test] + fn schema_rejects_execution_attribution_physical_backfill() { + let db = temp_db("executed-input-physical-backfill"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let mut head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + storage + .append_user_ops_chunk(&mut head, &[pending_user_op(0), pending_user_op(1)]) + .expect("append physical-only user ops"); + let physical_offsets = physical_user_op_offsets(&storage); + storage + .conn + .execute( + "INSERT INTO executed_inputs \ + (sequenced_l2_tx_offset, executed_input_offset) VALUES (?1, 0)", + [physical_offsets[1]], + ) + .expect("seed the later physical attribution"); + + let err = storage + .conn + .execute( + "INSERT INTO executed_inputs \ + (sequenced_l2_tx_offset, executed_input_offset) VALUES (?1, 1)", + [physical_offsets[0]], + ) + .expect_err("execution attribution cannot move backward physically"); + assert!( + err.to_string() + .contains("must follow physical replay order"), + "unexpected trigger error: {err:?}" + ); + } + + #[test] + fn schema_rejects_noncanonical_execution_offset() { + let db = temp_db("executed-input-logical-gap"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let mut head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + storage + .append_user_ops_chunk(&mut head, &[pending_user_op(0)]) + .expect("append physical-only user op"); + let physical_offset = physical_user_op_offsets(&storage)[0]; + + let err = storage + .conn + .execute( + "INSERT INTO executed_inputs \ + (sequenced_l2_tx_offset, executed_input_offset) VALUES (?1, 1)", + [physical_offset], + ) + .expect_err("the first canonical offset must equal the zero base"); + assert!( + err.to_string().contains("must equal canonical next count"), + "unexpected trigger error: {err:?}" + ); + } + + #[test] + fn schema_rejects_deleting_valid_execution_attribution() { + let db = temp_db("executed-input-valid-delete"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let mut head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + storage + .append_executed_user_ops_chunk(&mut head, &[included_user_op(0, 0)]) + .expect("append mapped user op"); + + let err = storage + .conn + .execute( + "DELETE FROM executed_inputs WHERE executed_input_offset = 0", + [], + ) + .expect_err("valid canonical attribution is not deletable"); + assert!( + err.to_string() + .contains("valid executed input attribution cannot be deleted"), + "unexpected trigger error: {err:?}" + ); + assert_eq!( + storage.next_executed_input_count().expect("preserved head"), + ExecutedInputCount::new(1) + ); + } + + #[test] + fn live_direct_rotation_requires_complete_classified_attribution() { + let db = temp_db("direct-execution-attribution-complete"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + pin_deployment_identity(&mut storage, SENDER_A); + let mut head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + let directs = [ + StoredSafeInput { + sender: Address::ZERO, + payload: vec![0xaa], + block_number: 10, + }, + StoredSafeInput { + sender: Address::repeat_byte(0x44), + payload: vec![0xbb], + block_number: 10, + }, + ]; + storage + .append_safe_inputs(10, &directs, SENDER_A, &default_protocol_timing()) + .expect("persist direct inputs"); + + let incomplete = [DirectInputExecution { + safe_input_index: 0, + executed_input_offset: ExecutedInputCount::ZERO, + }]; + let panic = catch_unwind(AssertUnwindSafe(|| { + let _ = storage.close_frame_only_with_executions( + &mut head, + 10, + SafeInputRange::new(0, 2), + &incomplete, + ); + })); + assert!(panic.is_err(), "omitted executable direct must fail loud"); + assert_eq!(head.frame_in_batch, 0); + let frames: i64 = storage + .conn + .query_row("SELECT COUNT(*) FROM frames", [], |row| row.get(0)) + .unwrap(); + assert_eq!(frames, 1, "failed rotation must roll back its new frame"); + + let complete = [ + DirectInputExecution { + safe_input_index: 0, + executed_input_offset: ExecutedInputCount::ZERO, + }, + DirectInputExecution { + safe_input_index: 1, + executed_input_offset: ExecutedInputCount::new(1), + }, + ]; + storage + .close_frame_only_with_executions(&mut head, 10, SafeInputRange::new(0, 2), &complete) + .expect("commit complete direct attribution"); + assert_eq!( + storage.next_executed_input_count().unwrap(), + ExecutedInputCount::new(2) + ); + } + + #[test] + fn invalidation_rewinds_and_replacement_reuses_logical_offset() { + let db = temp_db("execution-offset-invalidation-reuse"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let mut head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + + storage + .append_executed_user_ops_chunk(&mut head, &[included_user_op(0, 0)]) + .expect("append gold input"); + storage + .close_frame_and_batch(&mut head, 0) + .expect("seal gold batch"); + storage + .append_executed_user_ops_chunk(&mut head, &[included_user_op(1, 1)]) + .expect("append doomed input"); + assert_eq!( + storage + .next_executed_input_count() + .expect("pre-recovery count"), + ExecutedInputCount::new(2) + ); + + storage + .append_safe_inputs(1_500, &[], SENDER_A, &default_protocol_timing()) + .expect("advance safe head"); + assert_eq!( + storage + .recover_aging_tip(1_200) + .expect("invalidate stale Tip"), + vec![1] + ); + assert_eq!( + storage.next_executed_input_count().expect("rewound count"), + ExecutedInputCount::new(1) + ); + + let mut replacement = storage.open_state().unwrap().unwrap(); + storage + .append_executed_user_ops_chunk(&mut replacement, &[included_user_op(1, 1)]) + .expect("reuse invalidated logical offset"); + + let current: Vec = storage + .conn + .prepare( + "SELECT executed_input_offset FROM executed_inputs \ + ORDER BY sequenced_l2_tx_offset", + ) + .unwrap() + .query_map([], |row| row.get(0)) + .unwrap() + .collect::>>() + .unwrap(); + assert_eq!(current, vec![0, 1]); + let valid: Vec = storage + .conn + .prepare( + "SELECT executed_input_offset FROM valid_executed_inputs \ + ORDER BY sequenced_l2_tx_offset", + ) + .unwrap() + .query_map([], |row| row.get(0)) + .unwrap() + .collect::>>() + .unwrap(); + assert_eq!(valid, vec![0, 1]); + let physical_rows: i64 = storage + .conn + .query_row("SELECT COUNT(*) FROM sequenced_l2_txs", [], |row| { + row.get(0) + }) + .unwrap(); + assert_eq!( + physical_rows, 3, + "invalidation removes only derived mappings, not physical audit rows" + ); + } + + #[test] + fn snapshot_promotion_preserves_executed_input_count() { + let db = temp_db("snapshot-executed-input-count-promotion"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let mut head = storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + storage + .append_executed_user_ops_chunk(&mut head, &[included_user_op(0, 0)]) + .expect("append mapped user op"); + let physical_head = storage.valid_ordered_l2_tx_head().unwrap(); + storage + .insert_pending_dump(&db._dir.path().join("pending"), 0, physical_head) + .expect("insert pending checkpoint"); + assert_eq!( + storage + .latest_pending_dump() + .unwrap() + .unwrap() + .executed_input_count, + ExecutedInputCount::new(1) + ); + + storage + .promote_finalized(0, 10) + .expect("promote checkpoint"); + assert_eq!( + storage + .finalized_dump() + .unwrap() + .unwrap() + .executed_input_count, + ExecutedInputCount::new(1), + "promotion must copy the application boundary with the physical cursor" + ); + } + + #[test] + fn cockroach_padding_stays_unmapped_above_absolute_history_base() { + let db = temp_db("cockroach-padding-execution-offsets"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Rebuild) + .expect("initialize rebuild storage"); + let recovered = [ + StoredSafeInput { + sender: Address::ZERO, + payload: vec![0xaa], + block_number: 10, + }, + StoredSafeInput { + sender: Address::ZERO, + payload: vec![0xbb], + block_number: 10, + }, + ]; + storage + .append_safe_inputs(10, &recovered, SENDER_A, &default_protocol_timing()) + .expect("persist recovered L1 prefix"); + storage + .open_recovery_tip(10) + .expect("open padded recovery root"); + let physical_head = storage + .valid_ordered_l2_tx_head() + .expect("physical replay head"); + storage + .insert_initial_finalized_dump( + &db._dir.path().join("recovered"), + 10, + physical_head, + 41, + 2, + ) + .expect("bind recovered application base"); + + let mapped: i64 = storage + .conn + .query_row("SELECT COUNT(*) FROM executed_inputs", [], |row| row.get(0)) + .unwrap(); + assert_eq!(mapped, 0, "recovery-root padding is never re-attributed"); + assert_eq!( + storage + .next_executed_input_count() + .expect("absolute next count"), + ExecutedInputCount::new(41) + ); + let replay = storage.ordered_l2_txs_page_from(0, 10).unwrap(); + assert_eq!(replay.len(), 2); + assert!(replay.iter().all(|row| row.executed_input_offset.is_none())); + } + + #[test] + fn recovery_checkpoint_block_uses_query_and_persistence_boundaries() { + let db = temp_db("recovery-checkpoint-boundaries"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + storage + .append_safe_inputs( + 10, + &[StoredSafeInput { + sender: Address::ZERO, + payload: vec![], + block_number: 10, + }], + SENDER_A, + &default_protocol_timing(), + ) + .expect("append safe input"); + + let boundary = storage + .read(|tx| super::safe_input_index_exclusive_through_block_in(tx, u64::MAX)) + .expect("query upper bound above SQLite range"); + assert_eq!( + boundary, 1, + "a saturated upper predicate must include every representable block" + ); + + let err = storage + .open_recovery_tip(i64::MAX as u64 + 1) + .expect_err("unrepresentable persisted checkpoint must be refused"); + assert!(matches!(err, rusqlite::Error::ToSqlConversionFailure(_))); + } + #[test] fn next_frame_fee_comes_from_batch_policy() { let db = temp_db("batch-policy-fee"); @@ -828,11 +1633,11 @@ mod tests { .ordered_l2_txs_page_from(0, 100) .expect("load replay"); assert_eq!(replay.len(), 2); - match &replay[0].1 { + match &replay[0].tx { SequencedL2Tx::Direct(value) => assert_eq!(value.payload.as_slice(), &[0xaa]), _ => panic!("expected direct input at position 0"), } - match &replay[1].1 { + match &replay[1].tx { SequencedL2Tx::Direct(value) => assert_eq!(value.payload.as_slice(), &[0xbb]), _ => panic!("expected direct input at position 1"), } @@ -842,6 +1647,7 @@ mod tests { fn ensure_open_tip_opens_genesis_and_sequences_leading_range() { let db = temp_db("ensure-tip-genesis"); let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + pin_deployment_identity(&mut storage, SENDER_A); // Pre-existing L1 history: two safe inputs at block 10, and an // observed safe head of 10 (as the startup recovery sync would leave). @@ -891,8 +1697,9 @@ mod tests { "leading range [0,2) sequenced into genesis frame 0" ); - // Sequenced, not executed: the rows are in the ordered L2-tx stream - // for catch-up to replay, but ensure_open_tip ran no application code. + // The rows are in the ordered L2-tx stream for catch-up to replay and + // carry their creation-time application offsets. ensure_open_tip does + // not itself run application code; replay validates those offsets. let replay = storage .ordered_l2_txs_page_from(0, 100) .expect("load replay"); @@ -901,6 +1708,14 @@ mod tests { 2, "leading directs are in the replay stream for catch-up" ); + assert_eq!( + replay[0].executed_input_offset, + Some(ExecutedInputCount::ZERO) + ); + assert_eq!( + replay[1].executed_input_offset, + Some(ExecutedInputCount::new(1)) + ); } #[test] diff --git a/sequencer/src/storage/l1_inputs.rs b/sequencer/src/storage/l1_inputs.rs index b94e4b27..ac712d61 100644 --- a/sequencer/src/storage/l1_inputs.rs +++ b/sequencer/src/storage/l1_inputs.rs @@ -7,11 +7,15 @@ //! Also exposes the read-side queries the input reader and other callers need //! (current safe block, safe-input bounds, last safe-progress timestamp). -use alloy_primitives::{Address, B256}; +use alloy_primitives::Address; +#[cfg(test)] +use alloy_primitives::B256; use rusqlite::{OptionalExtension, Result, Transaction, params}; use super::Storage; -use super::convert::{i64_to_u64, now_unix_ms, u64_to_i64}; +use super::convert::{ + external_u64_to_i64, i64_to_u64, now_unix_ms, saturating_query_bound, u64_to_i64, +}; use super::queries::{ current_safe_block, current_safe_block_timestamp, last_safe_progress_ms, query_latest_safe_input_index_exclusive, @@ -31,56 +35,19 @@ type FeeOracleIdentitySql = ( Option, ); -trait SafeInputRecord { - fn sender(&self) -> Address; - fn payload(&self) -> &[u8]; - fn block_number(&self) -> u64; - fn block_timestamp(&self) -> u64; - fn transaction_hash(&self) -> B256; -} - -impl SafeInputRecord for StoredSafeInput { - fn sender(&self) -> Address { - self.sender - } - - fn payload(&self) -> &[u8] { - self.payload.as_slice() - } - - fn block_number(&self) -> u64 { - self.block_number - } - - // Synthetic storage fixtures do not model L1 provenance. - fn block_timestamp(&self) -> u64 { - 0 - } - - fn transaction_hash(&self) -> B256 { - B256::ZERO - } -} - -impl SafeInputRecord for IngestedSafeInput { - fn sender(&self) -> Address { - self.sender - } - - fn payload(&self) -> &[u8] { - self.payload.as_slice() - } - - fn block_number(&self) -> u64 { - self.block_number - } - - fn block_timestamp(&self) -> u64 { - self.block_timestamp - } - - fn transaction_hash(&self) -> B256 { - self.transaction_hash +/// Test-fixture conversion: a provenance-free [`StoredSafeInput`] becomes a +/// row with explicit zero provenance. Production writes go through +/// [`Storage::append_ingested_safe_inputs_with_timestamp`] with real +/// provenance only — one honest row model, no trait shim (H6; the Track 6 +/// inventory recorded the old `SafeInputRecord` shim as churn-avoidance). +#[cfg(test)] +fn synthetic_row(input: &StoredSafeInput) -> IngestedSafeInput { + IngestedSafeInput { + sender: input.sender, + payload: input.payload.clone(), + block_number: input.block_number, + block_timestamp: 0, + transaction_hash: B256::ZERO, } } @@ -105,7 +72,7 @@ impl Storage { /// to find any previous-instance batch past the checkpoint block. /// /// Queries the reader-synced `safe_inputs` table rather than issuing its own - /// `get_logs`, so it inherits the reader's F5 completeness guarantees (a + /// `get_logs`, so it inherits the reader's input-completeness witness (a /// per-app index contiguity check plus a `getNumberOfInputs` count witness): /// the reader refuses to persist an incomplete `get_logs` response, so the /// synced table is complete through the safe head. Do **not** replace this @@ -125,7 +92,10 @@ impl Storage { "SELECT safe_input_index, block_number FROM safe_inputs \ WHERE sender = ?1 AND block_number > ?2 \ ORDER BY safe_input_index ASC LIMIT 1", - params![batch_submitter.as_slice(), u64_to_i64(after_block)], + params![ + batch_submitter.as_slice(), + saturating_query_bound(after_block) + ], |row| { let index: i64 = row.get(0)?; let block: i64 = row.get(1)?; @@ -145,7 +115,7 @@ impl Storage { /// Returns both senders and directs; the caller classifies (a batch iff /// `sender == batch_submitter`) and drops batches from the `(A, B]` seed set. /// Read-only; queries the reader-synced table rather than a fresh log scan, - /// inheriting the reader's F5 completeness guarantees (see + /// inheriting the reader's input-completeness witness (see /// [`Storage::first_batch_submitter_input_after_block`]). pub fn safe_inputs_in_block_range( &mut self, @@ -157,7 +127,10 @@ impl Storage { ORDER BY safe_input_index ASC"; let mut stmt = self.conn.prepare_cached(SQL)?; let rows = stmt.query_map( - params![u64_to_i64(after_block), u64_to_i64(through_block)], + params![ + saturating_query_bound(after_block), + saturating_query_bound(through_block) + ], |row| { Ok(( row.get::<_, Vec>(0)?, @@ -178,21 +151,12 @@ impl Storage { Ok(out) } - /// Atomically: insert `inputs` (assigned contiguous indexes starting from - /// the current MAX+1), advance `l1_safe_head.block_number` to `safe_block`, - /// stamp `synced_at_ms` as the wall-clock time when the safe frontier - /// advanced, and update `safe_accepted_batches` via `protocol` so the - /// scheduler-accepted frontier view stays consistent with the safe head. - /// - /// The materialized `safe_accepted_batches` view is an invariant of this - /// operation: after a successful `append_safe_inputs`, every safe input up - /// to `safe_block` has been evaluated against the scheduler's acceptance - /// rules and recorded in `safe_accepted_batches`. Readers (submitter, - /// recovery, danger checks) never need to populate separately. - /// - /// Asserts `safe_block` is monotonic and that it strictly advances when - /// `inputs` is non-empty. - pub fn append_safe_inputs( + /// Test seed: [`Storage::append_safe_inputs_with_timestamp`] at the + /// current wall clock. See + /// [`Storage::append_ingested_safe_inputs_with_timestamp`] for the + /// operation's contract. + #[cfg(test)] + pub(crate) fn append_safe_inputs( &mut self, safe_block: u64, inputs: &[StoredSafeInput], @@ -210,13 +174,15 @@ impl Storage { } /// Same as [`Storage::append_safe_inputs`], but records the L1 timestamp - /// of `safe_block`. Synthetic inputs receive zero provenance; production - /// input-reader code uses [`Storage::append_ingested_safe_inputs_with_timestamp`]. + /// of `safe_block`. Test seed only: synthetic inputs receive explicit zero + /// provenance via [`synthetic_row`]; the production write path is + /// [`Storage::append_ingested_safe_inputs_with_timestamp`]. /// /// `frontier` gates the `safe_accepted_batches` update — see /// [`FrontierMode`]. Everything except `setup --recovery`'s interim syncs /// uses [`FrontierMode::Populate`]. - pub fn append_safe_inputs_with_timestamp( + #[cfg(test)] + pub(crate) fn append_safe_inputs_with_timestamp( &mut self, safe_block: u64, safe_block_timestamp: u64, @@ -225,18 +191,36 @@ impl Storage { timing: &ProtocolTiming, frontier: FrontierMode, ) -> Result<()> { - self.append_safe_input_records_with_timestamp( + let rows: Vec = inputs.iter().map(synthetic_row).collect(); + self.append_ingested_safe_inputs_with_timestamp( safe_block, safe_block_timestamp, - inputs, + rows.as_slice(), batch_submitter, timing, frontier, ) } - /// Production input-reader path. Persists per-input L1 provenance together - /// with the safe-input row and safe-head advance. + /// Production input-reader path — the one write of the safe-input stream. + /// + /// Atomically: insert `inputs` (assigned contiguous indexes starting from + /// the current MAX+1) with their per-input L1 provenance, advance + /// `l1_safe_head.block_number` to `safe_block`, stamp `synced_at_ms` as + /// the wall-clock time when the safe frontier advanced, and update + /// `safe_accepted_batches` via `timing` so the scheduler-accepted + /// frontier view stays consistent with the safe head. + /// + /// The materialized `safe_accepted_batches` view is an invariant of this + /// operation while no divergence exists: every safe input up to + /// `safe_block` has been evaluated against the scheduler's acceptance + /// rules. A foreign/mismatched accepted landing instead commits the + /// `canonical_divergence` fact with this head and freezes the projection; + /// later head advances remain paired with that terminal fact. Readers + /// (submitter, recovery, danger checks) never populate separately. + /// + /// Asserts `safe_block` is monotonic and that it strictly advances when + /// `inputs` is non-empty. pub(crate) fn append_ingested_safe_inputs_with_timestamp( &mut self, safe_block: u64, @@ -245,25 +229,6 @@ impl Storage { batch_submitter: Address, timing: &ProtocolTiming, frontier: FrontierMode, - ) -> Result<()> { - self.append_safe_input_records_with_timestamp( - safe_block, - safe_block_timestamp, - inputs, - batch_submitter, - timing, - frontier, - ) - } - - fn append_safe_input_records_with_timestamp( - &mut self, - safe_block: u64, - safe_block_timestamp: u64, - inputs: &[T], - batch_submitter: Address, - timing: &ProtocolTiming, - frontier: FrontierMode, ) -> Result<()> { self.write(|tx| { if let Some(current) = current_safe_block(tx)? { @@ -289,8 +254,8 @@ impl Storage { block_timestamp = excluded.block_timestamp, \ synced_at_ms = excluded.synced_at_ms", params![ - u64_to_i64(safe_block), - u64_to_i64(safe_block_timestamp), + external_u64_to_i64(safe_block, "L1 safe block")?, + external_u64_to_i64(safe_block_timestamp, "L1 safe block timestamp")?, now_unix_ms() ], )?; @@ -376,10 +341,13 @@ impl Storage { fee_oracle_pool, fee_oracle_twap_window_secs) \ VALUES (0, ?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11)", params![ - u64_to_i64(identity.chain_id), + external_u64_to_i64(identity.chain_id, "deployment chain ID")?, identity.app_address.as_slice(), identity.input_box_address.as_slice(), - u64_to_i64(identity.app_deployment_block), + external_u64_to_i64( + identity.app_deployment_block, + "application deployment block" + )?, identity.batch_submitter_address.as_slice(), mode, fixed, @@ -396,28 +364,9 @@ impl Storage { }) } - /// Record that `setup` finished. This is `setup`'s LAST write — after - /// identity is pinned, the initial L1 sync is durable, and the genesis - /// finalized snapshot is registered. - /// Idempotent: re-running `setup` on an already-complete DB leaves the - /// original `completed_at_ms` untouched. - pub fn mark_setup_complete(&mut self) -> Result<()> { - self.write(|tx| { - // Deliberate idempotency (re-running `setup` is legitimate), not - // silent absorption: keep the first completion timestamp. - tx.execute( - "INSERT INTO setup_complete (singleton_id, completed_at_ms) \ - VALUES (0, ?1) \ - ON CONFLICT(singleton_id) DO NOTHING", - params![now_unix_ms()], - )?; - Ok(()) - }) - } - /// Whether `setup` has completed on this DB. `run` refuses to boot when - /// this is `false` — the marker absent means either setup never ran or it - /// crashed midway, both of which require `setup` (re-)run, not `run`. + /// this is `false`: setup either never ran or crashed midway, both of + /// which require `setup` (re-)run, not `run`. pub fn is_setup_complete(&self) -> Result { let present: i64 = self.conn.query_row( "SELECT EXISTS(SELECT 1 FROM setup_complete WHERE singleton_id = 0)", @@ -428,7 +377,9 @@ impl Storage { } } -fn query_deployment_identity(conn: &rusqlite::Connection) -> Result> { +pub(super) fn query_deployment_identity( + conn: &rusqlite::Connection, +) -> Result> { conn.query_row( "SELECT chain_id, app_address, input_box_address, \ app_deployment_block, batch_submitter_address, fee_oracle_mode, \ @@ -463,10 +414,10 @@ fn query_deployment_identity(conn: &rusqlite::Connection) -> Result( +fn insert_safe_inputs_batch( tx: &Transaction<'_>, start_index: u64, - inputs: &[T], + inputs: &[IngestedSafeInput], ) -> Result<()> { if inputs.is_empty() { return Ok(()); @@ -477,13 +428,19 @@ fn insert_safe_inputs_batch( VALUES (?1, ?2, ?3, ?4, ?5, ?6)", )?; for (offset, input) in inputs.iter().enumerate() { + let offset = u64::try_from(offset) + .expect("safe-input batch offset exceeds u64: contract-impossible"); stmt.execute(params![ - u64_to_i64(start_index.saturating_add(offset as u64)), - input.sender().as_slice(), - input.payload(), - u64_to_i64(input.block_number()), - u64_to_i64(input.block_timestamp()), - input.transaction_hash().as_slice(), + u64_to_i64( + start_index + .checked_add(offset) + .expect("safe-input index overflow: contract-impossible"), + ), + input.sender.as_slice(), + input.payload.as_slice(), + external_u64_to_i64(input.block_number, "safe-input L1 block")?, + external_u64_to_i64(input.block_timestamp, "safe-input L1 timestamp")?, + input.transaction_hash.as_slice(), ])?; } Ok(()) @@ -492,11 +449,11 @@ fn insert_safe_inputs_batch( #[cfg(test)] mod tests { use crate::storage::{ - DeploymentIdentity, FeeOracleIdentity, FrontierMode, SafeInputRange, Storage, - StoredSafeInput, + DeploymentIdentity, FeeOracleIdentity, FrontierMode, IngestedSafeInput, SafeInputRange, + Storage, StoredSafeInput, test_helpers::{SENDER_A, SENDER_B, default_protocol_timing, temp_db}, }; - use alloy_primitives::Address; + use alloy_primitives::{Address, B256}; fn identity() -> DeploymentIdentity { DeploymentIdentity { @@ -611,6 +568,14 @@ mod tests { .expect("scan"), None ); + // Config accepts the full u64 range. Bounds above SQLite's INTEGER + // range are past every representable block and must not panic. + assert_eq!( + storage + .first_batch_submitter_input_after_block(SENDER_A, u64::MAX) + .expect("scan past SQLite range"), + None + ); } #[test] @@ -682,6 +647,21 @@ mod tests { .len(), 4 ); + assert_eq!( + storage + .safe_inputs_in_block_range(0, u64::MAX) + .expect("all through saturated upper bound") + .len(), + 4, + "an upper bound above SQLite's range includes every stored row" + ); + assert!( + storage + .safe_inputs_in_block_range(u64::MAX, u64::MAX) + .expect("empty past SQLite range") + .is_empty(), + "a lower bound above SQLite's range excludes every stored row" + ); } #[test] @@ -691,8 +671,13 @@ mod tests { assert_eq!(storage.batch_tree_anchor().expect("default"), 0); storage.set_batch_tree_anchor(1200).expect("set anchor"); assert_eq!(storage.batch_tree_anchor().expect("read back"), 1200); - // Once setup is complete, the public setter aborts too (write-once). - storage.mark_setup_complete().expect("mark complete"); + storage + .conn + .execute( + "INSERT INTO setup_complete (singleton_id, completed_at_ms) VALUES (0, 1)", + [], + ) + .expect("seed setup completion fact"); assert!( storage.set_batch_tree_anchor(1300).is_err(), "anchor must be frozen after setup_complete" @@ -726,6 +711,110 @@ mod tests { ); } + #[test] + fn external_values_outside_sqlite_integer_range_are_typed_refusals() { + let db = temp_db("external-integer-range"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let protocol = default_protocol_timing(); + + let safe_head_err = storage + .append_safe_inputs_with_timestamp( + u64::MAX, + 1, + &[], + SENDER_A, + &protocol, + FrontierMode::Populate, + ) + .expect_err("unrepresentable provider safe block must be refused"); + assert!(matches!( + safe_head_err, + rusqlite::Error::ToSqlConversionFailure(_) + )); + let safe_timestamp_err = storage + .append_safe_inputs_with_timestamp( + 10, + u64::MAX, + &[], + SENDER_A, + &protocol, + FrontierMode::Populate, + ) + .expect_err("unrepresentable provider safe timestamp must be refused"); + assert!(matches!( + safe_timestamp_err, + rusqlite::Error::ToSqlConversionFailure(_) + )); + + let input_err = storage + .append_safe_inputs( + 10, + &[StoredSafeInput { + sender: SENDER_B, + payload: vec![], + block_number: u64::MAX, + }], + SENDER_A, + &protocol, + ) + .expect_err("unrepresentable provider input block must be refused"); + assert!(matches!( + input_err, + rusqlite::Error::ToSqlConversionFailure(_) + )); + let input_timestamp_err = storage + .append_ingested_safe_inputs_with_timestamp( + 10, + 1, + &[IngestedSafeInput { + sender: SENDER_B, + payload: vec![], + block_number: 10, + block_timestamp: u64::MAX, + transaction_hash: B256::ZERO, + }], + SENDER_A, + &protocol, + FrontierMode::Populate, + ) + .expect_err("unrepresentable provider input timestamp must be refused"); + assert!(matches!( + input_timestamp_err, + rusqlite::Error::ToSqlConversionFailure(_) + )); + assert_eq!( + storage.current_safe_block().expect("read safe head"), + None, + "failed external writes must roll back atomically" + ); + assert_eq!( + storage.safe_input_end_exclusive().expect("read input head"), + 0 + ); + + let identity_err = storage + .load_or_insert_deployment_identity(DeploymentIdentity { + chain_id: u64::MAX, + ..identity() + }) + .expect_err("unrepresentable configured chain ID must be refused"); + assert!(matches!( + identity_err, + rusqlite::Error::ToSqlConversionFailure(_) + )); + let deployment_block_err = storage + .load_or_insert_deployment_identity(DeploymentIdentity { + app_deployment_block: u64::MAX, + ..identity() + }) + .expect_err("unrepresentable configured application deployment block must be refused"); + assert!(matches!( + deployment_block_err, + rusqlite::Error::ToSqlConversionFailure(_) + )); + assert_eq!(storage.deployment_identity().expect("read identity"), None); + } + #[test] fn deployment_identity_is_inserted_once() { let db = temp_db("deployment-identity-insert-once"); @@ -768,48 +857,6 @@ mod tests { ); } - #[test] - fn setup_complete_marker_absent_until_marked_then_idempotent() { - let db = temp_db("setup-complete-marker"); - let mut storage = Storage::open(db.path.as_str()).expect("open storage"); - - assert!( - !storage - .is_setup_complete() - .expect("read marker on fresh DB"), - "fresh DB has no setup-complete marker" - ); - - storage.mark_setup_complete().expect("mark complete"); - assert!( - storage.is_setup_complete().expect("read marker"), - "marker present after mark_setup_complete" - ); - - let first_ts: i64 = storage - .conn - .query_row( - "SELECT completed_at_ms FROM setup_complete WHERE singleton_id = 0", - [], - |row| row.get(0), - ) - .expect("read completed_at_ms"); - - // Re-running setup is legitimate and must not error or move the - // original timestamp. - storage.mark_setup_complete().expect("mark complete again"); - let second_ts: i64 = storage - .conn - .query_row( - "SELECT completed_at_ms FROM setup_complete WHERE singleton_id = 0", - [], - |row| row.get(0), - ) - .expect("read completed_at_ms"); - assert_eq!(first_ts, second_ts, "idempotent: first timestamp kept"); - assert!(storage.is_setup_complete().expect("read marker")); - } - #[test] fn append_safe_inputs_creates_and_advances_safe_head() { let db = temp_db("append-safe-inputs-creates-safe-head"); diff --git a/sequencer/src/storage/l1_submission.rs b/sequencer/src/storage/l1_submission.rs index 3e371e27..cd647673 100644 --- a/sequencer/src/storage/l1_submission.rs +++ b/sequencer/src/storage/l1_submission.rs @@ -4,7 +4,7 @@ //! The submitter's storage half: frontier lookup, per-batch frames + user //! ops, the catch-up / per-batch replay reader, the SSZ-encoded pending-batch //! list the submitter pulls each tick — and the one submission-side write, -//! the wallet-nonce watermark (raised before every broadcast, review R1a). +//! the wallet-nonce watermark (raised before every broadcast). //! //! Structural nonces are assigned by the `batches.nonce` trigger at close //! time (see `ingress`), and `safe_accepted_batches` is maintained by @@ -15,7 +15,7 @@ use rusqlite::{Result, params}; use super::Storage; -use super::convert::{i64_to_u16, i64_to_u32, i64_to_u64, u64_to_i64}; +use super::convert::{external_u64_to_i64, i64_to_u16, i64_to_u32, i64_to_u64, u64_to_i64}; use super::mutations::{batch_tree_anchor_in, set_batch_tree_anchor_in}; use super::queries::{current_safe_block_required, decode_l2_tx_row}; use super::safe_accepted_batches::frontier_nonce; @@ -33,9 +33,9 @@ impl Storage { /// /// **Precondition:** at least one safe-head observation must have been /// recorded (via [`Storage::append_safe_inputs`]). In production this is - /// always true because `run_preemptive_recovery` either syncs L1 first - /// or refuses to boot via `L1ViewStale`. Tests must seed an observation - /// explicitly; calling against a fresh DB returns `QueryReturnedNoRows`. + /// always true because the startup reducer completes InitialSync and + /// refuses admission unless its persisted view is usable. Tests must seed + /// an observation explicitly; a fresh DB returns `QueryReturnedNoRows`. pub fn submitter_frontier(&mut self) -> Result { self.read(|tx| { Ok(SubmitterFrontier { @@ -59,7 +59,7 @@ impl Storage { /// The highest wallet nonce ever broadcast by this deployment's /// batch-submitter key, or `None` if nothing was ever broadcast - /// (review R1a — the durable realization of the TLA+ `walletNonce`). + /// (the durable realization of the TLA+ `walletNonce`). /// The flush reads this as its coverage floor; it never resets. pub fn wallet_nonce_watermark(&mut self) -> Result> { use rusqlite::OptionalExtension; @@ -74,7 +74,7 @@ impl Storage { }) } - /// Write-before-broadcast (review R1a): durably raise the watermark to + /// Write-before-broadcast: durably raise the watermark to /// cover `nonce` *before* any tx at a nonce `<= nonce` is sent. The /// commit is power-loss durable (`synchronous=FULL`); a crash between /// commit and send only over-covers — the flush later no-ops a @@ -86,7 +86,7 @@ impl Storage { VALUES (0, ?1) \ ON CONFLICT(singleton_id) \ DO UPDATE SET watermark = MAX(watermark, excluded.watermark)", - params![u64_to_i64(nonce)], + params![external_u64_to_i64(nonce, "batch-submitter wallet nonce")?], )?; Ok(()) }) @@ -225,7 +225,7 @@ fn frames_for_batch_in(conn: &rusqlite::Connection, batch_index: u64) -> Result< /// Free-function form so the seal path can encode the closing batch inside /// its own transaction — the content-identity check's hash-at-seal must come -/// from **the same encode path the submitter uses** (review R2); this +/// from **the same encode path the submitter uses**; this /// function being that single path is load-bearing. pub(super) fn load_batch_frames_in( conn: &rusqlite::Connection, @@ -429,6 +429,16 @@ mod tests { storage.raise_wallet_nonce_watermark(9).expect("raise to 9"); assert_eq!(storage.wallet_nonce_watermark().expect("read"), Some(9)); + + let err = storage + .raise_wallet_nonce_watermark(i64::MAX as u64 + 1) + .expect_err("external wallet nonce outside SQLite range must be refused"); + assert!(matches!(err, rusqlite::Error::ToSqlConversionFailure(_))); + assert_eq!( + storage.wallet_nonce_watermark().expect("read"), + Some(9), + "failed external raise must not alter the durable watermark" + ); } #[test] @@ -721,6 +731,170 @@ mod tests { ); } + #[test] + fn check_danger_refuses_when_safe_block_timestamp_is_in_the_future() { + let db = temp_db("check-danger-future-safe-timestamp"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let protocol = default_test_protocol(); + let now_ms = 1_000_000_u64; + + // A full block-time (12 s) ahead of the local clock: unusable. Sub-block + // ahead-ness is NTP-scale skew and tolerated (see the companion assert). + storage + .append_safe_inputs_with_timestamp( + 10, + 1_012, + &[], + SENDER_A, + &protocol, + crate::storage::FrontierMode::Populate, + ) + .expect("record future L1 timestamp"); + storage + .conn + .execute( + "UPDATE l1_safe_head SET synced_at_ms = ?1 WHERE singleton_id = 0", + [i64::try_from(now_ms).expect("test time fits")], + ) + .expect("make local progress baseline usable"); + + assert_eq!( + storage + .check_danger(&protocol, now_ms) + .expect("check danger"), + crate::storage::DangerStatus::L1ViewStale + ); + + // Sub-block skew (1 s ahead) is the freshest possible view, not a fault. + storage + .conn + .execute( + "UPDATE l1_safe_head SET block_timestamp = 1001 WHERE singleton_id = 0", + [], + ) + .expect("set sub-block-ahead timestamp"); + assert_eq!( + storage + .check_danger(&protocol, now_ms) + .expect("check danger"), + crate::storage::DangerStatus::Safe + ); + } + + #[test] + fn check_danger_refuses_when_local_progress_timestamp_is_in_the_future() { + let db = temp_db("check-danger-future-progress"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let protocol = default_test_protocol(); + let now_ms = 1_000_000_u64; + + storage + .append_safe_inputs_with_timestamp( + 10, + 1_000, + &[], + SENDER_A, + &protocol, + crate::storage::FrontierMode::Populate, + ) + .expect("record safe head"); + // One full block-time (12 s) of regression: the wall-clock + // extrapolation is genuinely invalid — refuse. A sub-block step is + // tolerated (companion assert below). + storage + .conn + .execute( + "UPDATE l1_safe_head SET synced_at_ms = ?1 WHERE singleton_id = 0", + [i64::try_from(now_ms + 12_000).expect("test time fits")], + ) + .expect("move local progress baseline into future"); + + assert_eq!( + storage + .check_danger(&protocol, now_ms) + .expect("check danger"), + crate::storage::DangerStatus::L1ViewStale + ); + + // A 1 ms step is quantization noise for a block-granular estimate. + storage + .conn + .execute( + "UPDATE l1_safe_head SET synced_at_ms = ?1 WHERE singleton_id = 0", + [i64::try_from(now_ms + 1).expect("test time fits")], + ) + .expect("move baseline a sub-block step into the future"); + assert_eq!( + storage + .check_danger(&protocol, now_ms) + .expect("check danger"), + crate::storage::DangerStatus::Safe + ); + } + + #[test] + fn observed_batch_danger_outranks_regressed_progress_clock() { + let db = temp_db("check-danger-future-progress-priority"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let mut head = storage + .initialize_open_state(10, SafeInputRange::empty_at(0)) + .expect("initialize"); + storage + .close_frame_and_batch(&mut head, 10) + .expect("close batch 0"); + let protocol = default_test_protocol(); + let now_ms = 1_000_000_u64; + + storage + .append_safe_inputs_with_timestamp( + 1200, + 1_000, + &[], + SENDER_A, + &protocol, + crate::storage::FrontierMode::Populate, + ) + .expect("advance observed safe block past danger"); + // Even a full-block clock regression (a real fault, not noise) must + // not suppress the observed arm: batch age here is pure block + // arithmetic on persisted L1 observations, valid regardless of the + // local clock. The refusal only wins when no observed danger stands + // (2026-07-31 review of the containment commit). + storage + .conn + .execute( + "UPDATE l1_safe_head SET synced_at_ms = ?1 WHERE singleton_id = 0", + [i64::try_from(now_ms + 12_000).expect("test time fits")], + ) + .expect("move local progress baseline into future"); + + assert_eq!( + storage + .check_danger(&protocol, now_ms) + .expect("check danger"), + crate::storage::DangerStatus::ClosedBatchInDanger(0), + "observed danger stands on L1 observation alone; a wall-clock \ + fault must not delay recovery of a batch aging toward staleness" + ); + + // Same verdict with BOTH baselines faulted: the safe-block timestamp + // a full block ahead of the clock AND the progress baseline regressed. + storage + .conn + .execute( + "UPDATE l1_safe_head SET block_timestamp = ?1 WHERE singleton_id = 0", + [i64::try_from(now_ms / 1000 + 12).expect("test time fits")], + ) + .expect("move safe-block timestamp a block into the future"); + assert_eq!( + storage + .check_danger(&protocol, now_ms) + .expect("check danger"), + crate::storage::DangerStatus::ClosedBatchInDanger(0), + "both clock-fault baselines together still yield to observed danger" + ); + } + #[test] fn check_danger_safe_when_never_synced() { // Fresh DB, no prior safe block timestamp. The L1 view is unusable @@ -733,6 +907,28 @@ mod tests { assert_eq!(status, crate::storage::DangerStatus::L1ViewStale); } + #[test] + fn check_danger_errors_when_valid_tip_has_no_first_frame() { + let db = temp_db("check-danger-missing-first-frame"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + storage + .initialize_open_state(10, SafeInputRange::empty_at(0)) + .expect("initialize"); + let protocol = default_test_protocol(); + storage + .append_safe_inputs(10, &[], SENDER_A, &protocol) + .expect("record fresh safe head"); + storage + .conn + .execute("DELETE FROM frames WHERE batch_index = 0", []) + .expect("inject missing-frame corruption"); + + let err = storage + .check_danger(&protocol, unix_now_ms()) + .expect_err("a valid batch without a first frame must fail loud"); + assert!(matches!(err, rusqlite::Error::QueryReturnedNoRows)); + } + #[test] fn populate_safe_accepted_batches_resumes_from_latest_row() { let db = temp_db("safe-accepted-frontier-resume"); @@ -843,7 +1039,7 @@ mod tests { #[test] fn seal_stamps_payload_hash_of_the_submitter_encode_path() { - // Hash-at-seal (review R2): the hash stamped on the sealed row must + // Hash-at-seal: the hash stamped on the sealed row must // be the keccak256 of exactly the bytes the submitter will broadcast // (`pending_batches`'s encoding) — same code path, by construction. let db = temp_db("seal-stamps-payload-hash"); @@ -873,7 +1069,7 @@ mod tests { #[test] fn accepted_landing_with_mismatched_content_freezes_frontier() { - // The F1-zombie / F3-re-seal shape: a landing at the expected nonce + // The zombie / re-seal shape: a landing at the expected nonce // whose bytes differ from the batch we sealed. The check records a // 'mismatch' marker atomically with the sync and freezes the // frontier; later syncs stay frozen. @@ -939,7 +1135,7 @@ mod tests { // `scheduler_accepts` deliberately omits the two structural // rejections (future safe_block, non-monotonic frames) under // self-trust — the sequencer never produces them. Before the - // content-identity check (review R2), such a foreign batch at the + // content-identity check, such a foreign batch at the // expected nonce would be sim-accepted and silently desync the // frontier forever. With the check, it fails the local-batch lookup // (kind = foreign), the poison marker persists atomically with the diff --git a/sequencer/src/storage/lifecycle.rs b/sequencer/src/storage/lifecycle.rs new file mode 100644 index 00000000..d7b62031 --- /dev/null +++ b/sequencer/src/storage/lifecycle.rs @@ -0,0 +1,407 @@ +// (c) Cartesi and individual authors (see AUTHORS) +// SPDX-License-Identifier: Apache-2.0 (see LICENSE) + +//! The command-admission facts and the terminal-fault black box. +//! +//! Admission is governed by facts, each with one owner: +//! +//! - concurrent owners: the kernel process lock (`crate::runtime`); +//! - command ordering: `setup_complete`, checked two-sided here at every +//! preflight (setup/rebuild never restart over a completed setup; run and +//! maintenance never start before one); +//! - the one absorbing refusal: `canonical_divergence`, checked at every +//! entry — its only exit is cockroach rebuild; +//! - restart policy after a terminal fault: the exit-code contract +//! (30 = do not restart, page an operator), enforced by the supervisor, +//! not by a database gate. Standard recovery needs no intervention at +//! all: every run boots through the fact-derived recovery reducer. +//! +//! The black box (`terminal_faults`) is for operators and postmortems: the +//! cause of a terminal death, best-effort recorded before the process +//! exits, traveling with the data directory. Nothing reads it for +//! decisions, and its writes are verdict-neutral — a failed record loses +//! only the black-box copy; the exit code and logs still carry the verdict. + +use rusqlite::{OptionalExtension, TransactionBehavior, params}; +use thiserror::Error; + +use super::Storage; +use super::convert::{i64_to_u64, now_unix_ms}; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum LifecycleCommand { + Setup, + Rebuild, + Run, + MaintenanceFlush, +} + +impl LifecycleCommand { + pub(crate) const fn as_str(self) -> &'static str { + match self { + Self::Setup => "setup", + Self::Rebuild => "rebuild", + Self::Run => "run", + Self::MaintenanceFlush => "maintenance_flush", + } + } + + fn parse(value: &str) -> Result { + match value { + "setup" => Ok(Self::Setup), + "rebuild" => Ok(Self::Rebuild), + "run" => Ok(Self::Run), + "maintenance_flush" => Ok(Self::MaintenanceFlush), + other => Err(LifecycleError::Malformed(format!( + "unknown lifecycle command {other:?}" + ))), + } + } +} + +impl std::fmt::Display for LifecycleCommand { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(self.as_str()) + } +} + +/// One black-box row: which command died terminal, and why. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct TerminalFault { + pub command: LifecycleCommand, + pub cause: String, + pub recorded_at_ms: u64, +} + +#[derive(Debug, Error)] +pub enum LifecycleError { + #[error("lifecycle storage failed: {0}")] + Storage(#[from] rusqlite::Error), + /// A lifecycle contract was violated: a malformed black-box row, an + /// empty terminal cause, or setup completion attempted before its + /// preconditions exist. + #[error("lifecycle contract violated: {0}")] + Malformed(String), + #[error("{requested} is not admissible: {reason}")] + NotAdmissible { + requested: LifecycleCommand, + reason: &'static str, + }, + #[error( + "canonical divergence at batch nonce {nonce}; only cockroach recovery (fresh-directory rebuild) can proceed" + )] + CanonicalDivergence { nonce: u64 }, +} + +impl Storage { + /// The admission facts, checked read-only before a command does any + /// preparatory work: divergence is absorbing, and the two-sided + /// completion rule orders commands. + pub(crate) fn preflight_lifecycle_command( + &self, + command: LifecycleCommand, + ) -> Result<(), LifecycleError> { + refuse_on_canonical_divergence(&self.conn)?; + require_command_fits_completion(&self.conn, command) + } + + /// Commit setup's timeless completion fact. The `setup_complete` + /// primary key makes double-completion unrepresentable at the engine, + /// and the preconditions (finalized snapshot + application-history base) + /// are re-read inside the same transaction so completion can never + /// outrun the state it certifies. + pub(crate) fn complete_setup(&mut self) -> Result<(), LifecycleError> { + let tx = self + .conn + .transaction_with_behavior(TransactionBehavior::Immediate)?; + // Completing setup over persisted divergence would be a lie. + refuse_on_canonical_divergence(&tx)?; + let history = super::history::query_history_state(&tx)?; + let has_finalized_snapshot: bool = tx.query_row( + "SELECT EXISTS(SELECT 1 FROM finalized_snapshot WHERE singleton_id = 0)", + [], + |row| row.get(0), + )?; + if history.base_executed_input_count.is_none() + || history.base_safe_input_index.is_none() + || !has_finalized_snapshot + { + return Err(LifecycleError::Malformed( + "setup cannot complete before its finalized snapshot and \ + application-history base and safe-input floor are established" + .to_string(), + )); + } + tx.execute( + "INSERT INTO setup_complete (singleton_id, completed_at_ms) VALUES (0, ?1)", + [now_unix_ms()], + )?; + tx.commit()?; + Ok(()) + } + + /// Best-effort terminal-cause record. Deliberately gated on nothing — + /// not even divergence — because the recorder runs inside containment and + /// must never be the reason a cause goes unrecorded. Restart policy is + /// the exit-code contract, not this row. + pub(crate) fn record_terminal_fault( + &mut self, + command: LifecycleCommand, + cause: &str, + ) -> Result<(), LifecycleError> { + if cause.is_empty() { + return Err(LifecycleError::Malformed( + "terminal cause must not be empty".to_string(), + )); + } + self.conn.execute( + "INSERT INTO terminal_faults (command, cause, recorded_at_ms) \ + VALUES (?1, ?2, ?3)", + params![command.as_str(), cause, now_unix_ms()], + )?; + Ok(()) + } + + /// The most recent black-box row, or `None` when no terminal fault was + /// ever recorded. Operator/postmortem surface; nothing branches on it. + pub fn latest_terminal_fault(&self) -> Result, LifecycleError> { + let row = self + .conn + .query_row( + "SELECT command, cause, recorded_at_ms FROM terminal_faults \ + ORDER BY fault_id DESC LIMIT 1", + [], + |row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, String>(1)?, + row.get::<_, i64>(2)?, + )) + }, + ) + .optional()?; + let Some((command, cause, recorded_at_ms)) = row else { + return Ok(None); + }; + if cause.is_empty() { + return Err(LifecycleError::Malformed( + "terminal-fault row without cause".to_string(), + )); + } + Ok(Some(TerminalFault { + command: LifecycleCommand::parse(&command)?, + cause, + recorded_at_ms: u64::try_from(recorded_at_ms).map_err(|_| { + LifecycleError::Malformed(format!( + "terminal-fault recorded_at_ms {recorded_at_ms} is negative" + )) + })?, + })) + } +} + +/// The two-sided completion rule — the one command-ordering fact: +/// setup/rebuild never restart over a completed setup (completion is +/// once-per-database), run and maintenance never start before one exists. +fn require_command_fits_completion( + conn: &rusqlite::Connection, + command: LifecycleCommand, +) -> Result<(), LifecycleError> { + let setup_complete = setup_complete_exists(conn)?; + let (fits, reason) = match command { + LifecycleCommand::Setup | LifecycleCommand::Rebuild => ( + !setup_complete, + "setup is already complete for this data directory", + ), + LifecycleCommand::Run | LifecycleCommand::MaintenanceFlush => ( + setup_complete, + "setup has not completed for this data directory", + ), + }; + if !fits { + return Err(LifecycleError::NotAdmissible { + requested: command, + reason, + }); + } + Ok(()) +} + +fn setup_complete_exists(conn: &rusqlite::Connection) -> Result { + Ok(conn.query_row( + "SELECT EXISTS (SELECT 1 FROM setup_complete WHERE singleton_id = 0)", + [], + |row| row.get::<_, bool>(0), + )?) +} + +fn refuse_on_canonical_divergence(conn: &rusqlite::Connection) -> Result<(), LifecycleError> { + let nonce = conn + .query_row( + "SELECT nonce FROM canonical_divergence WHERE singleton_id = 0", + [], + |row| row.get::<_, i64>(0), + ) + .optional()?; + if let Some(nonce) = nonce { + return Err(LifecycleError::CanonicalDivergence { + nonce: i64_to_u64(nonce), + }); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::storage::test_helpers::temp_db; + + fn seeded(command: LifecycleCommand) -> (crate::storage::test_helpers::TestDb, Storage) { + let db = temp_db("lifecycle-facts"); + let storage = + Storage::initialize_for_command(db.path.as_str(), command).expect("initialize"); + (db, storage) + } + + fn complete_seeded_setup(storage: &mut Storage) { + storage + .insert_initial_finalized_dump(std::path::Path::new("/tmp/facts-genesis"), 0, 0, 0, 0) + .expect("register finalized snapshot"); + storage.complete_setup().expect("complete setup"); + } + + fn seed_divergence(storage: &Storage) { + storage + .conn + .execute( + "INSERT INTO canonical_divergence \ + (singleton_id, nonce, safe_input_index, kind, detected_at_ms) \ + VALUES (0, 7, 8, 'foreign', 9)", + [], + ) + .expect("seed divergence"); + } + + #[test] + fn completion_rule_is_two_sided() { + let (_db, mut storage) = seeded(LifecycleCommand::Rebuild); + // Before completion: run and maintenance refuse; rebuild may retry. + assert!(matches!( + storage.preflight_lifecycle_command(LifecycleCommand::Run), + Err(LifecycleError::NotAdmissible { .. }) + )); + assert!(matches!( + storage.preflight_lifecycle_command(LifecycleCommand::MaintenanceFlush), + Err(LifecycleError::NotAdmissible { .. }) + )); + storage + .preflight_lifecycle_command(LifecycleCommand::Rebuild) + .expect("rebuild retry over its incomplete database"); + + complete_seeded_setup(&mut storage); + // After completion: setup/rebuild refuse; run and maintenance pass. + assert!(matches!( + storage.preflight_lifecycle_command(LifecycleCommand::Setup), + Err(LifecycleError::NotAdmissible { .. }) + )); + assert!(matches!( + storage.preflight_lifecycle_command(LifecycleCommand::Rebuild), + Err(LifecycleError::NotAdmissible { .. }) + )); + storage + .preflight_lifecycle_command(LifecycleCommand::Run) + .expect("run passes over a completed setup"); + storage + .preflight_lifecycle_command(LifecycleCommand::MaintenanceFlush) + .expect("flush passes"); + } + + #[test] + fn divergence_refuses_every_preflight_and_setup_completion() { + let (_db, mut storage) = seeded(LifecycleCommand::Setup); + storage + .insert_initial_finalized_dump(std::path::Path::new("/tmp/facts-div"), 0, 0, 0, 0) + .expect("register finalized snapshot"); + seed_divergence(&storage); + + for command in [ + LifecycleCommand::Setup, + LifecycleCommand::Rebuild, + LifecycleCommand::Run, + LifecycleCommand::MaintenanceFlush, + ] { + assert!(matches!( + storage.preflight_lifecycle_command(command), + Err(LifecycleError::CanonicalDivergence { nonce: 7 }) + )); + } + assert!(matches!( + storage.complete_setup(), + Err(LifecycleError::CanonicalDivergence { nonce: 7 }) + )); + } + + #[test] + fn terminal_cause_recording_is_gated_on_nothing() { + let (_db, mut storage) = seeded(LifecycleCommand::Setup); + seed_divergence(&storage); + + storage + .record_terminal_fault( + LifecycleCommand::Run, + "persistent storage invariant violation", + ) + .expect("the containment recorder must always be able to write"); + let fault = storage + .latest_terminal_fault() + .expect("read") + .expect("recorded fault"); + assert_eq!(fault.command, LifecycleCommand::Run); + assert_eq!(fault.cause, "persistent storage invariant violation"); + assert!(matches!( + storage.record_terminal_fault(LifecycleCommand::Run, ""), + Err(LifecycleError::Malformed(_)) + )); + } + + #[test] + fn black_box_is_append_only_at_the_engine_and_reads_newest_first() { + let (_db, mut storage) = seeded(LifecycleCommand::Setup); + assert_eq!(storage.latest_terminal_fault().expect("empty read"), None); + storage + .record_terminal_fault(LifecycleCommand::Setup, "first death") + .expect("record first"); + storage + .record_terminal_fault(LifecycleCommand::MaintenanceFlush, "second death") + .expect("record second"); + let fault = storage + .latest_terminal_fault() + .expect("read") + .expect("latest fault"); + assert_eq!(fault.command, LifecycleCommand::MaintenanceFlush); + assert_eq!(fault.cause, "second death"); + + assert!( + storage + .conn + .execute("DELETE FROM terminal_faults", []) + .is_err() + ); + assert!( + storage + .conn + .execute("UPDATE terminal_faults SET cause = 'x'", []) + .is_err() + ); + } + + #[test] + fn setup_cannot_complete_before_base_and_snapshot_exist() { + let (_db, mut storage) = seeded(LifecycleCommand::Rebuild); + assert!(matches!( + storage.complete_setup(), + Err(LifecycleError::Malformed(_)) + )); + assert!(!storage.is_setup_complete().expect("read completion")); + } +} diff --git a/sequencer/src/storage/migrations/0001_schema.sql b/sequencer/src/storage/migrations/0001_schema.sql index 2127444c..3032e9da 100644 --- a/sequencer/src/storage/migrations/0001_schema.sql +++ b/sequencer/src/storage/migrations/0001_schema.sql @@ -23,19 +23,18 @@ -- wall clock — write-once (triggers below), but deliberately NOT -- cross-checked against `created_at_ms`: wall-clock monotonicity is an -- environmental assumption, not an invariant (NTP steps, VM resume), and a --- CHECK on it wedged batch close and the recovery cascade in a clock --- regression (review F8). No production code reads these as values; every +-- CHECK on it once wedged batch close and the recovery cascade in a clock +-- regression. No production code reads these as values; every -- reader is an IS NULL / IS NOT NULL predicate. -- `payload_hash` is the keccak256 of the batch's SSZ wire bytes, stamped at --- seal time by the same encode path the submitter uses (review R2, --- hash-at-seal). It is what the content-identity check compares an accepted +-- seal time by the same encode path the submitter uses (hash-at-seal). It is what the content-identity check compares an accepted -- L1 landing against — and because it is computed by the code that sealed -- the batch, it survives wire-format upgrades. NULL only while the batch is -- the open Tip (and on recovery sentinels, which carry no payload). CREATE TABLE IF NOT EXISTS batches ( batch_index INTEGER PRIMARY KEY, parent_batch_index INTEGER REFERENCES batches(batch_index), -- NULL only for genesis - nonce INTEGER NOT NULL CHECK (nonce >= 0), + nonce INTEGER NOT NULL CHECK (typeof(nonce) = 'integer' AND nonce >= 0), created_at_ms INTEGER NOT NULL, sealed_at_ms INTEGER CHECK (sealed_at_ms IS NULL OR sealed_at_ms >= 0), invalidated_at_ms INTEGER CHECK (invalidated_at_ms IS NULL OR invalidated_at_ms >= 0), @@ -96,6 +95,13 @@ INSERT OR IGNORE INTO batch_tree_anchor(singleton_id, nonce) VALUES (0, 0); -- would break it. The Rust writer is still the source of truth for the -- transition sequence — triggers just ensure the DB never reaches an -- inconsistent state if the writer misbehaves. +-- +-- typeof() guards: INTEGER affinity does not reject an unconvertible value, +-- and TEXT/BLOB sort above INTEGER, so a bare `x >= 0` CHECK passes 'abc'. +-- Columns that feed SQL-level arithmetic or comparisons (trigger math, +-- frontier folds, lease counts) therefore carry an explicit +-- typeof(x) = 'integer' guard: a mis-bound positional parameter must refuse, +-- not coerce to 0 inside a trigger. -- Nonce contiguity: `nonce = parent.nonce + 1`, or the batch-tree anchor nonce -- (0 for a genesis deployment, N' for a recovered one) for the parentless root. @@ -251,7 +257,8 @@ CREATE TABLE IF NOT EXISTS safe_inputs ( sender BLOB NOT NULL CHECK (length(sender) = 20), payload BLOB NOT NULL, -- Block number of the chain block where this direct input was included (e.g. InputAdded event block). - block_number INTEGER NOT NULL CHECK (block_number >= 0), + block_number INTEGER NOT NULL + CHECK (typeof(block_number) = 'integer' AND block_number >= 0), -- Timestamp of the carrying L1 block. block_timestamp INTEGER NOT NULL CHECK (block_timestamp >= 0), -- Hash of the L1 transaction that carried this input. @@ -328,7 +335,9 @@ WHERE batch_index NOT IN (SELECT batch_index FROM batches WHERE invalidated_at_m -- acceptance logic over new safe_inputs rows. CREATE TABLE IF NOT EXISTS safe_accepted_batches ( safe_input_index INTEGER PRIMARY KEY REFERENCES safe_inputs(safe_input_index), - nonce INTEGER NOT NULL, + -- CHECK aligns this column with its siblings (batches.nonce, anchor nonce); + -- the writer is u64-sourced, so a negative value is corruption. + nonce INTEGER NOT NULL CHECK (typeof(nonce) = 'integer' AND nonce >= 0), first_frame_safe_block INTEGER NOT NULL, inclusion_block INTEGER NOT NULL ); @@ -345,19 +354,19 @@ CREATE TABLE IF NOT EXISTS l1_safe_head ( ); -- Highest wallet nonce ever broadcast by this deployment's batch-submitter --- key (review R1a — the durable realization of the TLA+ spec's --- `walletNonce`). Write-before-broadcast: any component about to send a tx +-- key (the durable realization of the TLA+ spec's `walletNonce`). Write-before-broadcast: any component about to send a tx -- at wallet nonce n first commits watermark = max(watermark, n) — power-loss -- durable under synchronous=FULL — then sends. Uniform for batch txs and -- flush no-ops alike, so the flush's slot coverage never depends on the --- local node's volatile mempool memory (the F1 zombie). Absent row = +-- local node's volatile mempool memory (which a dropped-locally but +-- network-alive zombie tx evades). Absent row = -- nothing ever broadcast. Never reset, never lowered. CREATE TABLE IF NOT EXISTS wallet_nonce_watermark ( singleton_id INTEGER PRIMARY KEY CHECK (singleton_id = 0), watermark INTEGER NOT NULL CHECK (watermark >= 0) ); --- Canonical-divergence poison marker (review R2). Written by the input +-- Canonical-divergence poison marker. Written by the input -- reader's acceptance simulation — atomically with the sync that detected -- it — when a fully-accepted L1 landing fails the content-identity check: -- either no valid closed local batch exists at the accepted nonce @@ -376,6 +385,267 @@ CREATE TABLE IF NOT EXISTS canonical_divergence ( detected_at_ms INTEGER NOT NULL CHECK (detected_at_ms >= 0) ); +-- I15 structural enforcement: while the divergence marker exists, the batch +-- tree, promotions, and the pending-snapshot pool are frozen in the engine +-- itself. Standard recovery is forbidden on a diverged frontier; the typed +-- Rust refusals (the local-first startup reducer plus guarded Tip/Cascade +-- mutations and atomic runtime admission) remain the friendly error surface, but these +-- triggers are the enforcement a forgotten call site cannot bypass. +CREATE TRIGGER IF NOT EXISTS trg_batches_frozen_on_divergence_insert +BEFORE INSERT ON batches FOR EACH ROW +WHEN EXISTS (SELECT 1 FROM canonical_divergence WHERE singleton_id = 0) +BEGIN SELECT RAISE(ABORT, 'batch tree frozen: canonical divergence marker present'); END; + +CREATE TRIGGER IF NOT EXISTS trg_batches_frozen_on_divergence_update +BEFORE UPDATE ON batches FOR EACH ROW +WHEN EXISTS (SELECT 1 FROM canonical_divergence WHERE singleton_id = 0) +BEGIN SELECT RAISE(ABORT, 'batch tree frozen: canonical divergence marker present'); END; + +-- External history identity. One database serves exactly one era. The era is +-- minted with the baseline schema; standard recovery advances only the +-- generation. A rebuild's application-history base and durable safe-input +-- drain floor are unknown until the recovered finalized snapshot exists, so +-- they alone start NULL and fill together exactly once before setup completes. +CREATE TABLE IF NOT EXISTS history_state ( + singleton_id INTEGER PRIMARY KEY CHECK (singleton_id = 0), + era_id BLOB NOT NULL CHECK ( + typeof(era_id) = 'blob' + AND length(era_id) = 16 + AND substr(hex(era_id), 13, 1) = '4' + AND substr(hex(era_id), 17, 1) IN ('8', '9', 'A', 'B') + ), + era_created_at_ms INTEGER NOT NULL CHECK ( + typeof(era_created_at_ms) = 'integer' + AND era_created_at_ms >= 0 + ), + recovery_generation INTEGER NOT NULL CHECK ( + typeof(recovery_generation) = 'integer' + AND recovery_generation >= 0 + ), + base_executed_input_count INTEGER CHECK ( + base_executed_input_count IS NULL + OR ( + typeof(base_executed_input_count) = 'integer' + AND base_executed_input_count >= 0 + ) + ), + base_safe_input_index INTEGER CHECK ( + base_safe_input_index IS NULL + OR ( + typeof(base_safe_input_index) = 'integer' + AND base_safe_input_index >= 0 + ) + ), + CHECK ( + (base_executed_input_count IS NULL AND base_safe_input_index IS NULL) + OR + (base_executed_input_count IS NOT NULL AND base_safe_input_index IS NOT NULL) + ) +); + +CREATE TRIGGER IF NOT EXISTS trg_history_state_single_insert +BEFORE INSERT ON history_state +FOR EACH ROW +WHEN EXISTS (SELECT 1 FROM history_state WHERE singleton_id = 0) +BEGIN + SELECT RAISE(ABORT, 'history state is inserted once per database'); +END; + +CREATE TRIGGER IF NOT EXISTS trg_history_identity_write_once +BEFORE UPDATE OF singleton_id, era_id, era_created_at_ms ON history_state +FOR EACH ROW +BEGIN + SELECT RAISE(ABORT, 'history era identity is write-once'); +END; + +CREATE TRIGGER IF NOT EXISTS trg_history_base_write_once +BEFORE UPDATE OF base_executed_input_count, base_safe_input_index ON history_state +FOR EACH ROW +WHEN OLD.base_executed_input_count IS NOT NULL + OR OLD.base_safe_input_index IS NOT NULL +BEGIN + SELECT RAISE(ABORT, 'history base is write-once'); +END; + +CREATE TRIGGER IF NOT EXISTS trg_history_generation_monotonic +BEFORE UPDATE OF recovery_generation ON history_state +FOR EACH ROW +WHEN OLD.recovery_generation = 9223372036854775807 + OR NEW.recovery_generation != OLD.recovery_generation + 1 +BEGIN + SELECT RAISE(ABORT, 'recovery generation must advance by exactly one'); +END; + +CREATE TRIGGER IF NOT EXISTS trg_history_state_not_deletable +BEFORE DELETE ON history_state +FOR EACH ROW +BEGIN + SELECT RAISE(ABORT, 'history state is write-once per database'); +END; + +-- Canonical application-history coordinates attached to physical replay rows. +-- +-- `sequenced_l2_txs.offset` remains the append-only SQLite pagination cursor: +-- it may contain invalidated rows and rows that the application never executes +-- (our own batch submissions and cockroach-root cursor padding). This table is +-- the separate, sparse attribution saying which physical rows did execute and +-- at which `Application::executed_input_count` boundary. +-- +-- The primary key makes attribution one-to-one per physical row. This is a +-- derived *current canonical projection*, not the audit log: invalidating a +-- batch atomically deletes its mappings while retaining the physical replay +-- rows. The replacement suffix can then reuse its canonical offsets, enforced +-- by the global logical UNIQUE constraint. +CREATE TABLE IF NOT EXISTS executed_inputs ( + sequenced_l2_tx_offset INTEGER PRIMARY KEY + REFERENCES sequenced_l2_txs(offset), + executed_input_offset INTEGER NOT NULL CHECK ( + typeof(executed_input_offset) = 'integer' + AND executed_input_offset >= 0 + ), + UNIQUE(executed_input_offset) +); + +-- Invalidation structurally deletes mappings below, so this projection can +-- join the physical table directly without re-running the valid-batch filter. +CREATE VIEW IF NOT EXISTS valid_executed_inputs AS +SELECT + e.sequenced_l2_tx_offset, + e.executed_input_offset, + s.batch_index, + s.frame_in_batch, + s.user_op_pos_in_frame, + s.safe_input_index +FROM executed_inputs e +JOIN sequenced_l2_txs s ON s.offset = e.sequenced_l2_tx_offset; + +-- Attribution is creation-time state, not a catch-up repair operation. The +-- Rust writer maps rows in their creation transaction; this backstop limits a +-- target to the current valid Tip and refuses physical-order rewrites. +CREATE TRIGGER IF NOT EXISTS trg_executed_inputs_target_must_be_tip +BEFORE INSERT ON executed_inputs +FOR EACH ROW +WHEN NOT EXISTS ( + SELECT 1 + FROM sequenced_l2_txs s + JOIN batches b ON b.batch_index = s.batch_index + WHERE s.offset = NEW.sequenced_l2_tx_offset + AND b.sealed_at_ms IS NULL + AND b.invalidated_at_ms IS NULL +) +BEGIN + SELECT RAISE(ABORT, 'executed input must target the current valid Tip'); +END; + +CREATE TRIGGER IF NOT EXISTS trg_executed_inputs_requires_bound_base +BEFORE INSERT ON executed_inputs +FOR EACH ROW +WHEN (SELECT base_executed_input_count FROM history_state WHERE singleton_id = 0) IS NULL +BEGIN + SELECT RAISE(ABORT, 'executed input history base is not bound'); +END; + +CREATE TRIGGER IF NOT EXISTS trg_executed_inputs_physical_order +BEFORE INSERT ON executed_inputs +FOR EACH ROW +WHEN EXISTS ( + SELECT 1 FROM executed_inputs + WHERE sequenced_l2_tx_offset >= NEW.sequenced_l2_tx_offset +) +BEGIN + SELECT RAISE(ABORT, 'executed input attributions must follow physical replay order'); +END; + +-- Every new mapping consumes exactly the current canonical next offset: +-- max(the era base, MAX(current mapping) + 1). Invalidation deletes its suffix +-- mappings, naturally rewinding the next offset for the replacement suffix. +CREATE TRIGGER IF NOT EXISTS trg_executed_inputs_contiguous +BEFORE INSERT ON executed_inputs +FOR EACH ROW +WHEN NEW.executed_input_offset != ( + SELECT MAX( + base_executed_input_count, + COALESCE((SELECT MAX(executed_input_offset) + 1 FROM executed_inputs), 0) + ) + FROM history_state + WHERE singleton_id = 0 +) +BEGIN + SELECT RAISE(ABORT, 'executed input offset must equal canonical next count'); +END; + +CREATE TRIGGER IF NOT EXISTS trg_executed_inputs_append_only_update +BEFORE UPDATE ON executed_inputs +FOR EACH ROW +BEGIN + SELECT RAISE(ABORT, 'executed input attribution is append-only'); +END; + +CREATE TRIGGER IF NOT EXISTS trg_protect_valid_executed_input_delete +BEFORE DELETE ON executed_inputs +FOR EACH ROW +WHEN EXISTS ( + SELECT 1 + FROM sequenced_l2_txs s + JOIN batches b ON b.batch_index = s.batch_index + WHERE s.offset = OLD.sequenced_l2_tx_offset + AND b.invalidated_at_ms IS NULL +) +BEGIN + SELECT RAISE(ABORT, 'valid executed input attribution cannot be deleted'); +END; + +-- Recovery owns the only deletion path. The batch row is already invalid when +-- this AFTER trigger runs, so the guarded delete above permits exactly these +-- derived mappings to disappear in the same transaction as suffix invalidation. +CREATE TRIGGER IF NOT EXISTS trg_drop_invalidated_executed_inputs +AFTER UPDATE OF invalidated_at_ms ON batches +FOR EACH ROW +WHEN OLD.invalidated_at_ms IS NULL AND NEW.invalidated_at_ms IS NOT NULL +BEGIN + DELETE FROM executed_inputs + WHERE sequenced_l2_tx_offset IN ( + SELECT offset FROM sequenced_l2_txs WHERE batch_index = NEW.batch_index + ); +END; + +-- Terminal-fault black box: an append-only trail of terminal causes, +-- best-effort recorded before death. DELIBERATELY NOT AN ADMISSION GATE: +-- admission +-- is governed by facts — the kernel process lock excludes concurrent +-- owners, `setup_complete` orders commands (two-sided), and +-- `canonical_divergence` is the one absorbing refusal (cockroach rebuild is +-- the only exit). Restart policy after a terminal fault is the exit-code +-- contract (30 = do not restart, page), not a database gate. Nothing reads +-- this table for decisions; it exists so the cause of death travels with +-- the data directory for operator postmortems, surviving log rotation. +CREATE TABLE IF NOT EXISTS terminal_faults ( + fault_id INTEGER PRIMARY KEY AUTOINCREMENT, + command TEXT NOT NULL CHECK (command IN ( + 'setup', 'rebuild', 'run', 'maintenance_flush' + )), + cause TEXT NOT NULL CHECK ( + typeof(cause) = 'text' AND length(cause) > 0 + ), + recorded_at_ms INTEGER NOT NULL CHECK ( + typeof(recorded_at_ms) = 'integer' AND recorded_at_ms >= 0 + ) +); + +CREATE TRIGGER IF NOT EXISTS trg_terminal_faults_append_only_update +BEFORE UPDATE ON terminal_faults +FOR EACH ROW +BEGIN + SELECT RAISE(ABORT, 'terminal-fault black box is append-only'); +END; + +CREATE TRIGGER IF NOT EXISTS trg_terminal_faults_append_only_delete +BEFORE DELETE ON terminal_faults +FOR EACH ROW +BEGIN + SELECT RAISE(ABORT, 'terminal-fault black box is append-only'); +END; + -- Deployment identity: the persisted DB is only valid for this deployment. -- Allows L1-unreachable startup after first boot, and prevents interpreting -- historical sequencer state under a different app or batch-submitter address. @@ -472,7 +742,7 @@ CREATE TABLE IF NOT EXISTS batch_policy ( -- Log-space fee exponent fed by the oracle. log_gas_price INTEGER NOT NULL CHECK (log_gas_price >= 0), -- Unix-ms of the last successful oracle (or Fixed setup) write. - -- 0 means never written; Uniswap treats that as stale. + -- 0 means never written; completed setup guarantees a nonzero observation. log_gas_price_updated_at_ms INTEGER NOT NULL CHECK (log_gas_price_updated_at_ms >= 0), -- log_{129/128}(10), rounded by log_fee_ratio(10, 1) = 296. -- This tenfold price slack is applied in log space, not in the oracle. @@ -564,18 +834,43 @@ FROM batch_policy; CREATE TABLE IF NOT EXISTS dumps ( id INTEGER PRIMARY KEY, prefix TEXT NOT NULL UNIQUE, - lease_count INTEGER NOT NULL DEFAULT 0 CHECK (lease_count >= 0) + lease_count INTEGER NOT NULL DEFAULT 0 + CHECK (typeof(lease_count) = 'integer' AND lease_count >= 0) ); CREATE TABLE IF NOT EXISTS pending_snapshots ( - nonce INTEGER PRIMARY KEY CHECK (nonce >= 0), - dump_id INTEGER NOT NULL REFERENCES dumps(id) ON DELETE RESTRICT, - l2_tx_index INTEGER NOT NULL CHECK (l2_tx_index >= 0) + nonce INTEGER PRIMARY KEY CHECK (typeof(nonce) = 'integer' AND nonce >= 0), + dump_id INTEGER NOT NULL REFERENCES dumps(id) ON DELETE RESTRICT, + l2_tx_index INTEGER NOT NULL + CHECK (typeof(l2_tx_index) = 'integer' AND l2_tx_index >= 0), + executed_input_count INTEGER NOT NULL CHECK ( + typeof(executed_input_count) = 'integer' + AND executed_input_count >= 0 + ) ); CREATE TABLE IF NOT EXISTS finalized_snapshot ( - singleton_id INTEGER PRIMARY KEY CHECK (singleton_id = 0), - dump_id INTEGER NOT NULL REFERENCES dumps(id) ON DELETE RESTRICT, - inclusion_block INTEGER NOT NULL CHECK (inclusion_block >= 0), - l2_tx_index INTEGER NOT NULL CHECK (l2_tx_index >= 0) + singleton_id INTEGER PRIMARY KEY CHECK (singleton_id = 0), + dump_id INTEGER NOT NULL REFERENCES dumps(id) ON DELETE RESTRICT, + inclusion_block INTEGER NOT NULL + CHECK (typeof(inclusion_block) = 'integer' AND inclusion_block >= 0), + l2_tx_index INTEGER NOT NULL + CHECK (typeof(l2_tx_index) = 'integer' AND l2_tx_index >= 0), + executed_input_count INTEGER NOT NULL CHECK ( + typeof(executed_input_count) = 'integer' + AND executed_input_count >= 0 + ) ); + +-- I15 structural enforcement, snapshot half (batch-tree half lives next to +-- the canonical_divergence table): promotions and pending-pool clears are +-- frozen while the divergence marker exists. +CREATE TRIGGER IF NOT EXISTS trg_promotion_frozen_on_divergence +BEFORE INSERT ON finalized_snapshot FOR EACH ROW +WHEN EXISTS (SELECT 1 FROM canonical_divergence WHERE singleton_id = 0) +BEGIN SELECT RAISE(ABORT, 'promotion frozen: canonical divergence marker present'); END; + +CREATE TRIGGER IF NOT EXISTS trg_pending_clear_frozen_on_divergence +BEFORE DELETE ON pending_snapshots FOR EACH ROW +WHEN EXISTS (SELECT 1 FROM canonical_divergence WHERE singleton_id = 0) +BEGIN SELECT RAISE(ABORT, 'pending-snapshot clear frozen: canonical divergence marker present'); END; diff --git a/sequencer/src/storage/mod.rs b/sequencer/src/storage/mod.rs index db8e89c7..142eb7ef 100644 --- a/sequencer/src/storage/mod.rs +++ b/sequencer/src/storage/mod.rs @@ -16,6 +16,8 @@ //! - `admin` — operator policy alpha tuning //! - `fee_oracle` — L1 fee-oracle gas-price updates //! - `snapshot_dumps` — pending/finalized snapshot lifecycle, lease counts +//! - `history` — write-once era/base metadata and recovery generation +//! - `lifecycle` — command-admission facts + the terminal-fault black box //! //! Cross-writer helpers are split by concern: //! @@ -30,9 +32,11 @@ mod admin; mod convert; mod egress; mod fee_oracle; +mod history; mod ingress; mod l1_inputs; mod l1_submission; +mod lifecycle; mod mutations; mod open; mod queries; @@ -43,14 +47,21 @@ mod snapshot_dumps; #[cfg(test)] pub(crate) mod test_helpers; +pub(crate) use convert::is_persistent_storage_error; + use std::time::SystemTime; use thiserror::Error; pub(crate) use egress::OrderedL2TxRow; +pub use history::{DirectInputExecution, HistoryState}; +pub use lifecycle::{LifecycleCommand, LifecycleError, TerminalFault}; pub use open::Storage; pub use recovery::DangerStatus; +pub(crate) use recovery::{RecoveryInspection, RecoveryMutationError}; +pub use sequencer_core::history::{EraId, ExecutedInputCount, HistoryVersion, RecoveryGeneration}; pub use snapshot_dumps::{ - DumpRow, FinalizedDump, LeaseGuard, LeasedDump, PendingDump, ReleaseScheduler, + DumpRow, FinalizedDump, LeaseGuard, LeasedDump, PendingDump, PersistentReleaseFailureReporter, + ReleaseScheduler, }; /// One safe input as stored on the L1 InputBox: sender, opaque payload, and @@ -178,6 +189,14 @@ pub struct SafeInputFrontier { pub end_exclusive: u64, } +/// Whether the inclusion lane may reconcile the persisted L1 projection. +/// A poisoned projection never yields a usable frontier. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum SafeFrontierState { + Open(SafeInputFrontier), + CanonicalDivergence { nonce: u64, safe_input_index: u64 }, +} + /// Snapshot of the scheduler-accepted frontier: current safe block plus the /// next nonce the scheduler is expected to accept. Read by the batch submitter /// each tick to derive the next unresolved nonce. @@ -246,6 +265,28 @@ pub enum StorageOpenError { Sqlite(#[from] rusqlite::Error), #[error(transparent)] Migration(#[from] rusqlite_migration::Error), + /// No database file at the given path. Databases are only created by an + /// owning command (`setup` / `rebuild`); a missing file at `open` time is + /// a deployment mistake, never a cue to create one. + #[error( + "no database at {path} — this data directory was never initialized; run `setup` (or check --data-dir)" + )] + NeverInitialized { path: String }, +} + +pub(crate) fn is_persistent_storage_open_error(error: &StorageOpenError) -> bool { + match error { + StorageOpenError::Sqlite(source) => is_persistent_storage_error(source), + StorageOpenError::Migration(rusqlite_migration::Error::RusqliteError { err, .. }) => { + is_persistent_storage_error(err) + } + // Invalid schema versions/definitions and failed FK checks cannot be + // repaired by retrying the same durable database. + StorageOpenError::Migration(_) => true, + // A missing database is a deterministic deployment mistake; retrying + // the same configuration re-fails identically. + StorageOpenError::NeverInitialized { .. } => true, + } } /// Derived batch policy read from the `batch_policy_derived` view. @@ -258,9 +299,11 @@ pub struct BatchPolicy { pub batch_size_target: u16, } -/// In-memory mirror of the latest open batch + frame. Mutated by `Storage` -/// methods that change the open state (`append_user_ops_chunk`, `close_*`). -/// The lane keeps one `WriteHead` and threads it through every call. +/// Trusted, reconstructible in-memory cache of the latest durable open batch + +/// frame. Mutated by `Storage` methods that change the open state +/// (`append_executed_user_ops_chunk`, attributed `close_*`). The sole-writer +/// lane loads one from SQLite and threads it through every call; failed +/// writes/restarts discard it. Physical-only siblings are test fixtures. #[derive(Debug, Clone, Copy)] pub struct WriteHead { pub batch_index: u64, @@ -277,8 +320,20 @@ pub struct WriteHead { impl WriteHead { pub fn increment_batch_user_op_count(&mut self, count: usize) { - self.batch_user_op_count = self.batch_user_op_count.saturating_add(count as u64); - self.open_frame_user_op_count = self.open_frame_user_op_count.saturating_add(count as u32); + let count_u64 = + u64::try_from(count).expect("user-op chunk length exceeds u64: contract-impossible"); + let count_u32 = + u32::try_from(count).expect("user-op chunk length exceeds u32: contract-impossible"); + let next_batch_count = self + .batch_user_op_count + .checked_add(count_u64) + .expect("batch user-op count overflow: contract-impossible"); + let next_frame_count = self + .open_frame_user_op_count + .checked_add(count_u32) + .expect("frame user-op count overflow: contract-impossible"); + self.batch_user_op_count = next_batch_count; + self.open_frame_user_op_count = next_frame_count; } pub fn open_frame_has_user_ops(&self) -> bool { @@ -286,7 +341,10 @@ impl WriteHead { } pub fn advance_frame(&mut self, policy: BatchPolicy, safe_block: u64) { - self.frame_in_batch = self.frame_in_batch.saturating_add(1); + self.frame_in_batch = self + .frame_in_batch + .checked_add(1) + .expect("frame index overflow: contract-impossible"); self.frame_fee = policy.recommended_fee; self.safe_block = safe_block; self.open_frame_user_op_count = 0; @@ -314,6 +372,74 @@ impl WriteHead { /// Convert the log-space `batch_size_target` to a linear byte count for the inclusion lane. fn batch_size_target_bytes(policy: BatchPolicy) -> u64 { let linear = sequencer_core::fee::fee_to_linear(policy.batch_size_target); - // batch_size_target is always a reasonable byte count; clamp to u64. - linear.try_into().unwrap_or(u64::MAX) + linear.try_into().unwrap_or_else(|_| { + panic!( + "batch size target exponent {} does not fit u64 bytes: contract-impossible", + policy.batch_size_target + ) + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use alloy_primitives::U256; + use std::time::UNIX_EPOCH; + + fn write_head() -> WriteHead { + WriteHead { + batch_index: 0, + batch_created_at: UNIX_EPOCH, + frame_fee: 0, + safe_block: 0, + batch_user_op_count: 0, + open_frame_user_op_count: 0, + frame_in_batch: 0, + max_batch_user_op_bytes: 1, + } + } + + #[test] + #[should_panic(expected = "batch user-op count overflow: contract-impossible")] + fn write_head_batch_count_fails_loud_on_overflow() { + let mut head = write_head(); + head.batch_user_op_count = u64::MAX; + head.increment_batch_user_op_count(1); + } + + #[test] + #[should_panic(expected = "frame user-op count overflow: contract-impossible")] + fn write_head_frame_count_fails_loud_on_overflow() { + let mut head = write_head(); + head.open_frame_user_op_count = u32::MAX; + head.increment_batch_user_op_count(1); + } + + #[test] + #[should_panic(expected = "frame index overflow: contract-impossible")] + fn write_head_frame_index_fails_loud_on_overflow() { + let mut head = write_head(); + head.frame_in_batch = u32::MAX; + head.advance_frame( + BatchPolicy { + recommended_fee: 0, + batch_size_target: 0, + }, + 0, + ); + } + + #[test] + #[should_panic(expected = "does not fit u64 bytes: contract-impossible")] + fn batch_size_target_fails_loud_when_linear_value_exceeds_u64() { + let exponent = 6000; + assert!( + sequencer_core::fee::fee_to_linear(exponent) > U256::from(u64::MAX), + "test exponent must exceed the byte-counter representation" + ); + let _ = batch_size_target_bytes(BatchPolicy { + recommended_fee: 0, + batch_size_target: exponent, + }); + } } diff --git a/sequencer/src/storage/mutations.rs b/sequencer/src/storage/mutations.rs index 88dab7d2..04e90ca6 100644 --- a/sequencer/src/storage/mutations.rs +++ b/sequencer/src/storage/mutations.rs @@ -7,10 +7,15 @@ //! larger atomic unit. The two consumers today are ingress (batch/frame close //! + re-drain) and recovery (opening a recovery batch after cascade). +use alloy_primitives::Address; use rusqlite::{Connection, Result, Transaction, params}; -use super::SafeInputRange; use super::convert::{i64_to_u64, u64_to_i64}; +use super::history::{ + ExecutedInputMapping, attach_executed_inputs_in, next_executed_input_count_in, +}; +use super::l1_inputs::query_deployment_identity; +use super::{DirectInputExecution, SafeInputRange}; /// Insert a new batch. Nonce is derived from `parent_batch_index`: /// `parent.nonce + 1`, or 0 if `parent_batch_index` is None (genesis or @@ -72,7 +77,9 @@ fn compute_next_nonce(tx: &Transaction<'_>, parent_batch_index: Option) -> params![u64_to_i64(parent_bi)], |row| row.get(0), )?; - Ok(i64_to_u64(parent_nonce).saturating_add(1)) + Ok(i64_to_u64(parent_nonce) + .checked_add(1) + .expect("batch nonce overflow: contract-impossible")) } } } @@ -81,7 +88,7 @@ fn compute_next_nonce(tx: &Transaction<'_>, parent_batch_index: Option) -> /// `trg_sealed_at_ms_write_once` / `trg_payload_hash_write_once` triggers. /// The payload hash is stamped in the same UPDATE: a sealed batch always /// carries the hash the content-identity check compares accepted L1 -/// landings against (review R2, hash-at-seal). +/// landings against (hash-at-seal). pub(super) fn seal_batch( tx: &Transaction<'_>, batch_index: u64, @@ -128,7 +135,7 @@ pub(super) fn insert_open_frame( /// carries (0 for genesis, `N'` for a cockroach-recovered deployment). Composes /// inside the recovery-fill transaction. `trg_batch_tree_anchor_write_once` /// rejects this once `setup_complete` exists, so it is callable only during -/// setup, before the marker. +/// setup, before its atomic completion. pub(super) fn set_batch_tree_anchor_in(tx: &Transaction<'_>, nonce: u64) -> Result<()> { let changed = tx.execute( "UPDATE batch_tree_anchor SET nonce = ?1 WHERE singleton_id = 0", @@ -163,20 +170,155 @@ pub(super) fn persist_frame_direct_sequence( batch_index: u64, frame_in_batch: u32, range: SafeInputRange, + executions: &[DirectInputExecution], +) -> Result<()> { + let expected = derive_direct_input_executions_in(tx, range)?; + assert_eq!( + executions, expected, + "direct execution attributions must cover exactly the non-submitter drained rows from the canonical next count" + ); + persist_frame_direct_sequence_inner(tx, batch_index, frame_in_batch, range, executions) +} + +/// Sequence a production startup/recovery Tip's leading direct range, deriving +/// its complete application attribution from the persisted deployment identity +/// and current canonical application boundary. +pub(super) fn persist_frame_direct_sequence_derived( + tx: &Transaction<'_>, + batch_index: u64, + frame_in_batch: u32, + range: SafeInputRange, +) -> Result<()> { + let executions = derive_direct_input_executions_in(tx, range)?; + persist_frame_direct_sequence_inner(tx, batch_index, frame_in_batch, range, &executions) +} + +/// Sequence physical cursor rows that intentionally represent no newly +/// executed application inputs. Cockroach-root padding uses this because the +/// recovered snapshot already contains their effects; test fixtures use it +/// when exercising only the physical ordering layer. +pub(super) fn persist_frame_direct_sequence_physical_only( + tx: &Transaction<'_>, + batch_index: u64, + frame_in_batch: u32, + range: SafeInputRange, +) -> Result<()> { + persist_frame_direct_sequence_inner(tx, batch_index, frame_in_batch, range, &[]) +} + +fn persist_frame_direct_sequence_inner( + tx: &Transaction<'_>, + batch_index: u64, + frame_in_batch: u32, + range: SafeInputRange, + executions: &[DirectInputExecution], ) -> Result<()> { if range.is_empty() { + assert!( + executions.is_empty(), + "empty safe-input range cannot carry execution attributions" + ); return Ok(()); } + + let mut previous_safe_input_index = None; + for execution in executions { + assert!( + execution.safe_input_index >= range.start() && execution.safe_input_index < range.end(), + "direct execution attribution lies outside its drained range" + ); + if let Some(previous) = previous_safe_input_index { + assert!( + execution.safe_input_index > previous, + "direct execution attributions must be strictly ordered by safe-input index" + ); + } + previous_safe_input_index = Some(execution.safe_input_index); + } + let mut stmt = tx.prepare_cached( "INSERT INTO sequenced_l2_txs (batch_index, frame_in_batch, user_op_pos_in_frame, safe_input_index) \ VALUES (?1, ?2, NULL, ?3)", )?; + let mut executions = executions.iter().peekable(); + let mut mappings = Vec::with_capacity(executions.len()); for safe_input_index in range.start()..range.end() { stmt.execute(params![ u64_to_i64(batch_index), i64::from(frame_in_batch), u64_to_i64(safe_input_index), ])?; + if executions + .peek() + .is_some_and(|execution| execution.safe_input_index == safe_input_index) + { + let execution = executions.next().expect("peeked direct execution"); + mappings.push(ExecutedInputMapping { + sequenced_l2_tx_offset: i64_to_u64(tx.last_insert_rowid()), + executed_input_offset: execution.executed_input_offset, + }); + } } + assert!( + executions.next().is_none(), + "not every direct execution attribution was persisted" + ); + drop(stmt); + attach_executed_inputs_in(tx, &mappings)?; Ok(()) } + +fn derive_direct_input_executions_in( + tx: &Transaction<'_>, + range: SafeInputRange, +) -> Result> { + if range.is_empty() { + return Ok(Vec::new()); + } + + let identity = query_deployment_identity(tx)?.ok_or(rusqlite::Error::QueryReturnedNoRows)?; + let mut next = next_executed_input_count_in(tx)?; + let mut stmt = tx.prepare_cached( + "SELECT safe_input_index, sender \ + FROM safe_inputs \ + WHERE safe_input_index >= ?1 AND safe_input_index < ?2 \ + ORDER BY safe_input_index ASC", + )?; + let rows = stmt.query_map( + params![u64_to_i64(range.start()), u64_to_i64(range.end())], + |row| Ok((row.get::<_, i64>(0)?, row.get::<_, Vec>(1)?)), + )?; + + let mut executions = Vec::new(); + let mut fetched = 0_u64; + for row in rows { + let (safe_input_index, sender) = row?; + let safe_input_index = i64_to_u64(safe_input_index); + let expected_index = range + .start() + .checked_add(fetched) + .expect("safe-input classification index overflow: contract-impossible"); + assert_eq!( + safe_input_index, expected_index, + "non-contiguous safe-input range while assigning execution offsets" + ); + if Address::from_slice(sender.as_slice()) != identity.batch_submitter_address { + executions.push(DirectInputExecution { + safe_input_index, + executed_input_offset: next, + }); + next = next + .checked_next() + .expect("executed input count overflow: contract-impossible"); + } + fetched = fetched + .checked_add(1) + .expect("safe-input classification count overflow: contract-impossible"); + } + assert_eq!( + range.start().checked_add(fetched), + Some(range.end()), + "safe-input range was not fully populated while assigning execution offsets" + ); + Ok(executions) +} diff --git a/sequencer/src/storage/open.rs b/sequencer/src/storage/open.rs index 5e61bda7..b7579f88 100644 --- a/sequencer/src/storage/open.rs +++ b/sequencer/src/storage/open.rs @@ -6,35 +6,35 @@ //! Method clusters live in sibling files (`ingress`, `egress`, `l1_inputs`, //! `l1_submission`, `recovery`, `admin`) — each adds its own `impl Storage`. -use rusqlite::{Connection, OpenFlags, Result, Transaction, TransactionBehavior}; -use rusqlite_migration::{M, Migrations}; +use rusqlite::{Connection, OpenFlags, Result, Transaction, TransactionBehavior, types::Type}; +use rusqlite_migration::{HookResult, M, Migrations}; -use super::StorageOpenError; +use super::{EraId, LifecycleCommand, StorageOpenError}; const MIGRATION_0001_SCHEMA: &str = include_str!("migrations/0001_schema.sql"); /// SQLite `synchronous` pragma used by every production writer connection. /// `FULL` under WAL fsyncs on every commit, so commits survive power loss / -/// OS crash — not just process crash. Load-bearing (review R3/F3): the -/// sequencer externalizes effects on commits (acks `POST /tx` after the +/// OS crash — not just process crash. Load-bearing: the sequencer +/// externalizes effects on commits (acks `POST /tx` after the /// chunk commit; the submitter broadcasts sealed batches), and a rewound /// commit after externalization is silent divergence — e.g. a re-sealed /// batch at the same nonce with different content than the one the /// scheduler executed. The dump side already pays the same cost /// (`create_dump` fsyncs); this closes the DB half. Also a precondition -/// for the wallet-nonce watermark's write-before-broadcast guarantee -/// (review R1a). And it is what makes the `setup_complete` marker a valid -/// linearization point: the marker is committed in its own transaction -/// after the genesis-snapshot row's transaction, so "marker durable ⇒ +/// for the wallet-nonce watermark's write-before-broadcast guarantee. +/// And it is what makes the setup completion transaction a valid +/// linearization point: it commits after the genesis-snapshot row's +/// transaction, so "completion durable ⇒ /// snapshot row durable ⇒ dump dir durable" only holds because FULL fsyncs -/// every commit — under NORMAL the marker's WAL frame could survive while +/// every commit — under NORMAL the completion WAL frame could survive while /// the snapshot row's frames are lost, and `run` would boot a half-set-up /// DB. Benchmarked at the flip: round-trip/ack deltas were noise-level on -/// NVMe (see review ledger WP1). +/// NVMe. /// -/// Do not relax to NORMAL without revisiting all three (R3/F3 externalized -/// commits, R1a watermark, the setup-marker linearization in -/// `runtime/setup.rs` + `storage/migrations/0001_schema.sql`). +/// Do not relax to NORMAL without revisiting all three (externalized +/// commits, the write-before-broadcast watermark, and the setup-completion +/// linearization in `commands/setup/` + `storage/migrations/0001_schema.sql`). const SYNCHRONOUS_PRAGMA: &str = "FULL"; /// Sequencer storage backed by a single SQLite database. @@ -53,9 +53,44 @@ pub struct Storage { impl Storage { /// Production open: runs migrations, uses the canonical synchronous pragma. - pub fn open(path: &str) -> Result { + /// + /// Refuses a path with no database file (production builds): every + /// database is created by an owning command through + /// [`Storage::initialize_for_command`], so a missing file here is a + /// deployment mistake (mistyped `--data-dir`, wrong mount). Creating one + /// on the fly would mint an ownerless era with no creating command — + /// database absence means uninitialized, never create-and-proceed. + /// Crate tests keep create-on-open as their fixture idiom; the + /// command-less baseline in [`baseline_migration`] exists for them. + pub(crate) fn open(path: &str) -> Result { + #[cfg(not(test))] + if !std::path::Path::new(path).exists() { + return Err(StorageOpenError::NeverInitialized { + path: path.to_string(), + }); + } let mut conn = open_writer_connection(path)?; - run_migrations(&mut conn)?; + run_migrations(&mut conn, None)?; + Ok(Self { + conn, + path: path.to_string(), + }) + } + + /// Create the baseline schema and history era in one migration + /// transaction, with the creating command deciding the history bases. On + /// an already-migrated database the hook does not run; callers must + /// inspect the existing facts. + pub(crate) fn initialize_for_command( + path: &str, + command: LifecycleCommand, + ) -> Result { + assert!( + matches!(command, LifecycleCommand::Setup | LifecycleCommand::Rebuild), + "an uninitialized lifecycle may begin only with setup or rebuild" + ); + let mut conn = open_writer_connection(path)?; + run_migrations(&mut conn, Some(command))?; Ok(Self { conn, path: path.to_string(), @@ -153,7 +188,164 @@ fn open_reader_connection(path: &str) -> Result { /// Apply all migrations. Package-private — callers use [`Storage::open`] /// which runs this automatically. -pub(super) fn run_migrations(conn: &mut Connection) -> Result<(), StorageOpenError> { - Migrations::from_slice(&[M::up(MIGRATION_0001_SCHEMA)]).to_latest(conn)?; +pub(super) fn run_migrations( + conn: &mut Connection, + initial_command: Option, +) -> Result<(), StorageOpenError> { + let migration = baseline_migration(initial_command, None); + Migrations::from_slice(&[migration]).to_latest(conn)?; Ok(()) } + +type PostInitialMetadataHook = fn(&Transaction<'_>) -> HookResult; + +/// Build the exact baseline migration used in production. The optional hook +/// exists only so the atomicity test can fail *after* observing the +/// production history insert, without maintaining a shadow migration. +/// +/// `initial_command: None` is the crate-test fixture path (create-on-open +/// with genesis history bases); production cannot reach it because +/// [`Storage::open`] refuses paths with no database file. +fn baseline_migration( + initial_command: Option, + post_initial_metadata: Option, +) -> M<'static> { + M::up_with_hook(MIGRATION_0001_SCHEMA, move |tx: &Transaction<'_>| { + let recorded_at_ms = i64::try_from(crate::clock::unix_now_ms()).unwrap_or(i64::MAX); + let era_id = mint_era_id(tx)?; + let (base_executed_input_count, base_safe_input_index) = match initial_command { + Some(LifecycleCommand::Rebuild) => (None, None), + Some(LifecycleCommand::Setup) | None => (Some(0_i64), Some(0_i64)), + Some(LifecycleCommand::Run | LifecycleCommand::MaintenanceFlush) => { + unreachable!("baseline command was checked before migration") + } + }; + tx.execute( + "INSERT INTO history_state \ + (singleton_id, era_id, era_created_at_ms, recovery_generation, \ + base_executed_input_count, base_safe_input_index) \ + VALUES (0, ?1, ?2, 0, ?3, ?4)", + rusqlite::params![ + era_id.as_bytes().as_slice(), + recorded_at_ms, + base_executed_input_count, + base_safe_input_index + ], + )?; + if let Some(hook) = post_initial_metadata { + hook(tx)?; + } + Ok(()) + }) +} + +fn mint_era_id(tx: &Transaction<'_>) -> Result { + let random = tx.query_row("SELECT randomblob(16)", [], |row| row.get::<_, Vec>(0))?; + let mut bytes: [u8; EraId::BYTE_LEN] = random.try_into().map_err(|value: Vec| { + rusqlite::Error::FromSqlConversionFailure( + 0, + Type::Blob, + Box::new(std::io::Error::new( + std::io::ErrorKind::InvalidData, + format!( + "SQLite randomblob returned {} bytes, expected {}", + value.len(), + EraId::BYTE_LEN + ), + )), + ) + })?; + bytes[6] = (bytes[6] & 0x0f) | 0x40; + bytes[8] = (bytes[8] & 0x3f) | 0x80; + EraId::from_bytes(bytes) + .map_err(|error| rusqlite::Error::ToSqlConversionFailure(Box::new(error))) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn fail_after_observing_initial_metadata(tx: &Transaction<'_>) -> HookResult { + let history_count: i64 = tx.query_row( + "SELECT COUNT(*) FROM history_state \ + WHERE singleton_id = 0 AND recovery_generation = 0 \ + AND base_executed_input_count = 0 \ + AND base_safe_input_index = 0", + [], + |row| row.get(0), + )?; + if history_count != 1 { + return Err(rusqlite_migration::HookError::Hook( + "production initial history insert was not observed".to_string(), + )); + } + Err(rusqlite_migration::HookError::Hook( + "injected failure after initial history insert".to_string(), + )) + } + + #[test] + fn failing_initial_hook_rolls_back_schema_and_history_together() { + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("atomic-init.sqlite"); + let mut conn = open_writer_connection(path.to_str().expect("utf8")).expect("open"); + let definitions = [baseline_migration( + Some(LifecycleCommand::Setup), + Some(fail_after_observing_initial_metadata), + )]; + let migrations = Migrations::from_slice(&definitions); + migrations + .to_latest(&mut conn) + .expect_err("the injected hook failure must abort migration"); + + let table_count: i64 = conn + .query_row( + "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' \ + AND name NOT LIKE 'sqlite_%'", + [], + |row| row.get(0), + ) + .expect("inspect schema"); + let version: i64 = conn + .pragma_query_value(None, "user_version", |row| row.get(0)) + .expect("user version"); + assert_eq!(table_count, 0); + assert_eq!(version, 0); + } + + #[test] + fn baseline_mints_distinct_uuid_v4_eras_and_initializes_known_bases() { + let setup_dir = tempfile::tempdir().expect("setup tempdir"); + let setup_path = setup_dir.path().join("sequencer.sqlite"); + let setup = Storage::initialize_for_command( + setup_path.to_str().expect("utf8"), + LifecycleCommand::Setup, + ) + .expect("initialize setup"); + let setup_history = setup.history_state().expect("setup history"); + assert_eq!(setup_history.version.recovery_generation.get(), 0); + assert_eq!(setup_history.base_executed_input_count, Some(0)); + assert_eq!(setup_history.base_safe_input_index, Some(0)); + + let rebuild_dir = tempfile::tempdir().expect("rebuild tempdir"); + let rebuild_path = rebuild_dir.path().join("sequencer.sqlite"); + let rebuild = Storage::initialize_for_command( + rebuild_path.to_str().expect("utf8"), + LifecycleCommand::Rebuild, + ) + .expect("initialize rebuild"); + let rebuild_history = rebuild.history_state().expect("rebuild history"); + assert_eq!(rebuild_history.version.recovery_generation.get(), 0); + assert_eq!(rebuild_history.base_executed_input_count, None); + assert_eq!(rebuild_history.base_safe_input_index, None); + assert_ne!(setup_history.version.era_id, rebuild_history.version.era_id); + + let generic_dir = tempfile::tempdir().expect("generic tempdir"); + let generic_path = generic_dir.path().join("sequencer.sqlite"); + let generic = + Storage::open(generic_path.to_str().expect("utf8")).expect("initialize generic schema"); + let generic_history = generic.history_state().expect("generic history"); + assert_eq!(generic_history.base_executed_input_count, Some(0)); + assert_eq!(generic_history.base_safe_input_index, Some(0)); + } +} diff --git a/sequencer/src/storage/queries.rs b/sequencer/src/storage/queries.rs index e7abb822..55c0be97 100644 --- a/sequencer/src/storage/queries.rs +++ b/sequencer/src/storage/queries.rs @@ -97,7 +97,9 @@ pub(super) fn query_latest_safe_input_index_exclusive(conn: &Connection) -> Resu row.get(0) })?; Ok(match value { - Some(last_index) => i64_to_u64(last_index).saturating_add(1), + Some(last_index) => i64_to_u64(last_index) + .checked_add(1) + .expect("next safe-input index overflow: contract-impossible"), None => 0, }) } @@ -156,10 +158,34 @@ pub(super) fn query_batch_policy(conn: &Connection) -> Result { |row| Ok((row.get(0)?, row.get(1)?)), )?; let max_exp = sequencer_core::fee::MAX_EXPONENT; + // The two policy outputs have deliberately asymmetric legal ranges. + // Operator writes may push the recommended fee above the representable + // exponent (cap it), or the batch-size target below zero (floor it). + // Their opposite directions are contract-impossible and must fail loud. + let recommended_fee = if log_recommended_fee > i64::from(max_exp) { + max_exp + } else { + u16::try_from(log_recommended_fee).unwrap_or_else(|_| { + panic!( + "batch policy derived recommended fee {log_recommended_fee} is negative: \ + contract-impossible" + ) + }) + }; + let batch_size_target = if log_batch_size_target < 0 { + 0 + } else { + assert!( + log_batch_size_target <= i64::from(max_exp), + "batch policy derived batch size target {log_batch_size_target} exceeds \ + MAX_EXPONENT {max_exp}: contract-impossible" + ); + u16::try_from(log_batch_size_target) + .expect("batch size target checked within the u16 exponent range") + }; Ok(BatchPolicy { - // Clamp to MAX_EXPONENT to prevent panics in fee_to_linear. - recommended_fee: i64_to_u16(log_recommended_fee).min(max_exp), - batch_size_target: i64_to_u16(log_batch_size_target).min(max_exp), + recommended_fee, + batch_size_target, }) } diff --git a/sequencer/src/storage/recovery.rs b/sequencer/src/storage/recovery.rs index c8ebd9ac..9593619c 100644 --- a/sequencer/src/storage/recovery.rs +++ b/sequencer/src/storage/recovery.rs @@ -22,11 +22,12 @@ //! sequencer controls its own submissions — this is a deliberate system //! assumption, not a gap. -use rusqlite::{Connection, OptionalExtension, Result, Transaction, params}; +use rusqlite::{Connection, OptionalExtension, Result, Transaction, TransactionBehavior, params}; use sequencer_core::protocol::{ProtocolTiming, age_exceeds}; use super::Storage; use super::convert::{i64_to_u64, now_unix_ms, u64_to_i64}; +use super::history::advance_recovery_generation_in; use super::ingress::open_fresh_tip_in_tx; use super::queries::{ current_safe_block_required, current_safe_block_timestamp, last_safe_progress_ms, @@ -36,21 +37,19 @@ use super::snapshot_dumps::{batch_nonce_in, clear_pending_dumps_from_nonce_in}; /// Outcome of a danger-zone check. /// -/// Each variant maps to a distinct recovery response, encoded in -/// [`super::super::recovery::StartupAction`]: +/// Each variant maps to a distinct response in the startup recovery reducer: /// -/// - `L1ViewStale` → refuse boot. The L1 safe block is too old or unknown. -/// - `ClosedBatchInDanger(closed_idx)` → flush + cascade. A closed batch past the -/// accepted frontier has L1 transactions that may already be on chain; -/// we need the flush to resolve their fate before cascading. +/// - `L1ViewStale` → retry boot. The L1 safe block is too old or unknown. +/// - `ClosedBatchInDanger(closed_idx)` → enter the phase-granular +/// Flush/Sync/Cascade sequence. /// - `TipInDanger(tip_idx)` → direct Tip recovery, no flush. The Tip has no L1 /// footprint, so we can invalidate it and open a fresh one without /// any L1 round-trip. -/// - `EstimatedBatchInDanger(idx)` → refuse boot. The observed safe block is +/// - `EstimatedBatchInDanger(idx)` → retry boot. The observed safe block is /// still below the danger threshold, but wall-clock time since the last /// safe-head advance has consumed the batch's remaining runway. -/// - `Safe` → no recovery work; just ensure the Tip exists (torn-state -/// crash recovery branch). +/// - `Safe` → admit when a Tip exists, otherwise run `EnsureOpenTip` and +/// re-inspect. /// /// The runtime danger detector treats every non-`Safe` variant as /// "exit for recovery" — the difference between them only matters at the @@ -59,16 +58,16 @@ use super::snapshot_dumps::{batch_nonce_in, clear_pending_dumps_from_nonce_in}; pub enum DangerStatus { /// No danger detected — none of the checks tripped. Safe, - /// A fully-accepted L1 landing failed the content-identity check - /// (review R2): canonical state contains executed effects with no + /// A fully-accepted L1 landing failed the content-identity check: + /// canonical state contains executed effects with no /// reliable local source. Carries the diverged batch nonce. Ranked /// ahead of every other arm so the respawn loop can never route a - /// diverged node into `Proceed`/`FlushAndCascade`. The remedy is - /// cockroach recovery (wipe + rebuild from L1), never standard - /// recovery. + /// diverged node into a provider call, mutation, or admission. The remedy + /// is cockroach recovery (wipe + rebuild from L1), never standard recovery. CanonicalDivergence(u64), - /// L1 safe-head timestamp is too old or unknown. Recovery cannot reason - /// from the local L1 view, so startup must refuse. + /// L1 safe-head timestamp is too old/unknown, or the current clock + /// predates one of the persisted safety baselines. Recovery cannot reason + /// from the local L1 view, so startup must retry. L1ViewStale, /// Observed-safe check tripped on a *closed* batch past the /// accepted frontier: aged beyond `protocol.danger_threshold()` against @@ -86,6 +85,46 @@ pub enum DangerStatus { EstimatedBatchInDanger(u64), } +/// One transactionally consistent local view consumed by the startup +/// recovery reducer. +/// +/// Keeping these facts together is load-bearing: admission and recovery-phase +/// selection must not combine a danger verdict from one SQLite snapshot with +/// Tip/snapshot/head facts from another. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct RecoveryInspection { + pub(crate) danger: DangerStatus, + pub(crate) has_finalized_snapshot: bool, + pub(crate) has_open_tip: bool, + pub(crate) current_safe_block: Option, +} + +/// A recovery mutation was refused because the transaction no longer +/// satisfies the phase selected by the reducer. +#[derive(Debug, thiserror::Error)] +pub(crate) enum RecoveryMutationError { + #[error(transparent)] + Storage(#[from] rusqlite::Error), + #[error("canonical divergence at batch nonce {nonce} forbids standard recovery")] + CanonicalDivergence { nonce: u64 }, + #[error("recovery decision is stale: expected {expected:?}, found {actual:?}")] + StaleDecision { + expected: DangerStatus, + actual: DangerStatus, + }, + #[error("cannot open the Tip without a finalized snapshot")] + MissingFinalizedSnapshot, + #[error( + "post-flush re-sync reached safe block {resynced_safe_block}, behind the flush observation at {flush_observed_safe_block}" + )] + ResyncBehindFlushView { + resynced_safe_block: u64, + flush_observed_safe_block: u64, + }, + #[error("post-flush re-sync did not persist a safe head")] + MissingSafeHead, +} + impl DangerStatus { /// Stable label for logs/metrics. An inherent method (not a free /// projection) so a new variant must add its label right here. @@ -116,20 +155,40 @@ impl DangerStatus { } impl Storage { + /// Whether the canonical-divergence marker (I15) is present, and the + /// recorded `(nonce, safe_input_index)` if so. Standard + /// recovery is forbidden while the marker exists; callers on the recovery + /// path must check this before any batch-tree mutation. + pub fn canonical_divergence(&mut self) -> Result> { + self.read(|tx| canonical_divergence_in(tx)) + } + /// Unified danger-zone detection. /// - /// Runs four checks inside a single read transaction, in priority order: + /// Runs checks inside a single read transaction, in priority order: /// - /// 1. **L1 read freshness**: if the safe block timestamp is missing or + /// 1. **Canonical divergence**: an already-confirmed mismatch is an + /// absorbing terminal fact and outranks every view/clock condition. + /// 2. **L1 view freshness**: if the safe block timestamp is missing or /// older than `protocol.l1_read_stale_after_blocks`, return - /// `L1ViewStale`. A stale L1 view is unusable even if the RPC answers. - /// 2. **Observed closed-frontier**: `find_closed_frontier_batch_in_danger` + /// `L1ViewStale`. A stale L1 *view* is unusable even if the RPC + /// answers — recovery itself needs a trustworthy view, so this gate + /// stays ahead of everything. + /// 3. **Observed closed-frontier**: `find_closed_frontier_batch_in_danger` /// against `protocol.danger_threshold()`. Uses the observed safe block. - /// 3. **Observed open Tip**: `find_tip_batch_in_danger` against + /// 4. **Observed open Tip**: `find_tip_batch_in_danger` against /// `protocol.danger_threshold()`. Catches the case where all closed /// batches are gold but the Tip is aging — the lane is stuck or the /// Tip rotated without a safe-block advance. - /// 4. **Batch-relative wall-clock estimate**: if a correction applies + /// 5. **Clock faults**, deliberately after the observed arms: a local + /// clock a full block-time or more out of step with either persisted + /// baseline (behind the safe-block timestamp, or behind the local + /// last-progress baseline) is a *clock* fault, not a view fault. The + /// observed arms (3, 4) are pure block arithmetic, and a wall-clock + /// fault must never suppress a danger verdict that stands on L1 + /// observation alone. Sub-block skew in either direction is + /// quantization noise, not a fault. + /// 6. **Batch-relative wall-clock estimate**: if a correction applies /// ([`ProtocolTiming::wall_clock_adjusted_danger_threshold`] returns /// `Some`), widens to `find_first_batch_in_danger` against /// `danger_threshold − missed_blocks`. This is a fallback for when the @@ -138,13 +197,18 @@ impl Storage { /// to trust for continued soft confirmations. /// /// Returns the first variant that fires, in the order - /// `L1ViewStale` → `ClosedBatchInDanger` → `TipInDanger` → + /// `CanonicalDivergence` → `L1ViewStale` (stale view) → + /// `ClosedBatchInDanger` → `TipInDanger` → `L1ViewStale` (clock fault) → /// `EstimatedBatchInDanger` → `Safe`. The order encodes the /// "trust" hierarchy: /// - /// - **L1 view freshness gates everything.** If the safe block timestamp - /// is too old or unknown, neither recovery nor continued soft - /// confirmations are honest. + /// - **View staleness gates everything after divergence.** If the safe + /// block timestamp is too old or unknown, neither recovery nor + /// continued soft confirmations are honest. + /// - **Clock faults yield to observed danger.** A local clock a full + /// block-time or more out of step with either persisted baseline + /// refuses only when no observed danger stands; sub-block skew is + /// tolerated. /// - **Closed observed danger beats Tip.** When a closed batch is in danger, /// we need a flush (to resolve its L1 transaction's fate) regardless /// of the Tip's state. The cascade naturally catches the Tip via @@ -160,37 +224,102 @@ impl Storage { /// so the storage layer stays testable without time mocking. Production /// callers pass the current Unix-ms clock. pub fn check_danger(&mut self, protocol: &ProtocolTiming, now_ms: u64) -> Result { - self.read(|tx| { - // The divergence marker outranks everything — including the - // L1-staleness gate: it records an already-confirmed fact about - // canonical state, not a view-dependent estimate, and no amount - // of L1 freshening or flushing repairs it (review R2). - if let Some((nonce, _)) = canonical_divergence_in(tx)? { - return Ok(DangerStatus::CanonicalDivergence(nonce)); - } - - if protocol.l1_view_is_stale(current_safe_block_timestamp(tx)?, now_ms) { - return Ok(DangerStatus::L1ViewStale); - } - - let danger_threshold = protocol.danger_threshold(); - if let Some(idx) = find_closed_frontier_batch_in_danger(tx, danger_threshold)? { - return Ok(DangerStatus::ClosedBatchInDanger(idx)); - } - - if let Some(idx) = find_tip_batch_in_danger(tx, danger_threshold)? { - return Ok(DangerStatus::TipInDanger(idx)); - } - - let last = last_safe_progress_ms(tx)?; - if let Some(adjusted) = protocol.wall_clock_adjusted_danger_threshold(last, now_ms) - && let Some(idx) = find_first_batch_in_danger(tx, adjusted)? - { - return Ok(DangerStatus::EstimatedBatchInDanger(idx)); - } - - Ok(DangerStatus::Safe) - }) + self.read(|tx| check_danger_in(tx, protocol, now_ms)) + } + + /// Read every local fact used by the startup reducer in one transaction. + pub(crate) fn inspect_recovery( + &mut self, + protocol: &ProtocolTiming, + now_ms: u64, + ) -> Result { + self.read(|tx| inspect_recovery_in(tx, protocol, now_ms)) + } + + /// Execute the reducer's `EnsureOpenTip` phase only if its local decision + /// still holds in the write transaction. + pub(crate) fn ensure_open_tip_for_recovery( + &mut self, + protocol: &ProtocolTiming, + now_ms: u64, + ) -> std::result::Result<(), RecoveryMutationError> { + let tx = self + .conn + .transaction_with_behavior(TransactionBehavior::Immediate)?; + let facts = inspect_recovery_in(&tx, protocol, now_ms)?; + refuse_divergence(facts.danger)?; + if !facts.has_finalized_snapshot { + return Err(RecoveryMutationError::MissingFinalizedSnapshot); + } + if facts.danger != DangerStatus::Safe || facts.has_open_tip { + return Err(RecoveryMutationError::StaleDecision { + expected: DangerStatus::Safe, + actual: facts.danger, + }); + } + open_fresh_tip_in_tx(&tx)?; + tx.commit()?; + Ok(()) + } + + /// Execute the reducer's `RecoverTip` phase only while the same Tip is + /// still the observed-danger arm in the write transaction. + pub(crate) fn recover_aging_tip_for_recovery( + &mut self, + expected_batch_index: u64, + protocol: &ProtocolTiming, + now_ms: u64, + ) -> std::result::Result, RecoveryMutationError> { + let tx = self + .conn + .transaction_with_behavior(TransactionBehavior::Immediate)?; + let facts = inspect_recovery_in(&tx, protocol, now_ms)?; + refuse_divergence(facts.danger)?; + if !facts.has_finalized_snapshot { + return Err(RecoveryMutationError::MissingFinalizedSnapshot); + } + let expected = DangerStatus::TipInDanger(expected_batch_index); + if facts.danger != expected { + return Err(RecoveryMutationError::StaleDecision { + expected, + actual: facts.danger, + }); + } + let invalidated = recover_aging_tip_inner(&tx, protocol.danger_threshold())?; + tx.commit()?; + Ok(invalidated) + } + + /// Execute the reducer's `Cascade` phase. The ephemeral flush witness is + /// represented by its observed safe-block floor; this transaction + /// reasserts both I15 and the post-flush resync coherence check + /// immediately before changing the batch tree. + pub(crate) fn recover_post_flush_for_recovery( + &mut self, + flush_observed_safe_block: u64, + protocol: &ProtocolTiming, + now_ms: u64, + ) -> std::result::Result, RecoveryMutationError> { + let tx = self + .conn + .transaction_with_behavior(TransactionBehavior::Immediate)?; + let facts = inspect_recovery_in(&tx, protocol, now_ms)?; + refuse_divergence(facts.danger)?; + if !facts.has_finalized_snapshot { + return Err(RecoveryMutationError::MissingFinalizedSnapshot); + } + let resynced_safe_block = facts + .current_safe_block + .ok_or(RecoveryMutationError::MissingSafeHead)?; + if resynced_safe_block < flush_observed_safe_block { + return Err(RecoveryMutationError::ResyncBehindFlushView { + resynced_safe_block, + flush_observed_safe_block, + }); + } + let invalidated = recover_post_flush_inner(&tx, protocol.danger_threshold())?; + tx.commit()?; + Ok(invalidated) } /// Mark a single batch as invalid. Test-only seeder — production code goes @@ -207,127 +336,172 @@ impl Storage { Ok(()) } - /// Cascade everything past the gold frontier. Called from the - /// `FlushAndCascade` startup path, after the mempool flush has resolved - /// every wallet-nonce slot and `safe_accepted_batches` has been re-synced. - /// - /// # The "everything past gold is doomed" rule - /// - /// At this point the gold frontier is at its maximum extent: every - /// submitted batch has either been accepted (gold) or rejected by the - /// scheduler simulation (Silver-stale, since nonce-mismatch is impossible - /// at the frontier under self-trust), or its tx was killed by a flush - /// no-op (Pending, no `safe_input`). All three non-gold states are doomed: - /// - /// - **Silver-stale:** scheduler skipped it; downstream batches are - /// nonce-poisoned. - /// - **Pending:** the original L1 tx is dead. Re-submission could in - /// principle land fresh, but the *next* recovery cycle's flush would - /// compete with the resub at its new wallet-nonce slot and the bumped - /// no-op typically wins. The system would loop until current staleness - /// crossed `MAX_WAIT_BLOCKS`. Cascading now converges in one cycle. - /// - /// So once we've committed to recovery (the danger detector tripped, the - /// flush ran), the right move is to cascade the entire non-gold suffix - /// and open a fresh recovery batch. - /// - /// Three aftermath shapes: - /// - /// 1. **Everything worked:** all in-flight batches landed fresh and were - /// accepted. Gold extends to the last submitted batch; no first - /// non-gold closed. (See "Tip handling" below for the subtle subcase.) - /// 2. **Mixed:** some landed (stale or poisoned), some replaced. First - /// non-gold closed is either Silver-stale or Pending. Cascade from - /// there; the `batch_index >= N` rule catches the rest of the suffix - /// including the open Tip. - /// 3. **All replaced:** flush no-ops won every race. Gold doesn't - /// advance; first non-gold closed is the very first non-accepted batch. - /// - /// # Tip handling - /// - /// In cases (2)/(3) the cascade catches the Tip via `batch_index >= N`. - /// In case (1), there's no closed pivot — but the Tip can still be in - /// the danger zone: - /// - /// When the lane rotates a batch without a safe-block advance between - /// frames (e.g. immediately after init, when both share the bootstrap - /// `safe_block`), the Tip's `first_frame.safe_block` equals the closed - /// batch's. The closed batch can become gold by inclusion-staleness - /// (`inclusion_block - first_frame < MAX_WAIT`) while the Tip's age, - /// computed against `current_safe_block` after the flush wait, has - /// crossed `danger_threshold`. Pure monotonicity (`S_tip ≥ S_closed`) doesn't - /// rule this out — equality is allowed. - /// - /// So in the no-pivot branch we additionally check the Tip against - /// `danger_threshold` (the same threshold that would have triggered - /// recovery had the Tip been a closed batch). We're already committed - /// to recovery; the Tip is past gold; if it's also in the danger zone, - /// cascade it and open a fresh one. - /// - /// # Atomicity - /// - /// Runs as a single SQLite write transaction. On crash mid-way, the - /// txn rolls back; on commit, the cascade and the recovery batch open - /// land together. Idempotent on re-run because `valid_*` views filter - /// out already-invalidated rows. - /// - /// # Precondition - /// - /// The caller MUST have just synced L1 state via - /// [`Storage::append_safe_inputs`]; the gold frontier in - /// `safe_accepted_batches` must reflect the latest safe head. Otherwise - /// the cascade may invalidate batches that haven't yet had a chance to - /// be processed by the scheduler simulation. - /// - /// Returns the newly-invalidated batch indices (empty if none). + /// Test-only unguarded Cascade primitive. Production calls + /// [`Storage::recover_post_flush_for_recovery`]; the design rationale + /// lives on [`recover_post_flush_inner`], the shared body. + #[cfg(test)] pub fn recover_post_flush(&mut self, danger_threshold: u64) -> Result> { self.write(|tx| recover_post_flush_inner(tx, danger_threshold)) } - /// Cascade the open Tip if its first frame has aged past - /// `danger_threshold`. Called from the `RecoverTip` startup path (no flush - /// happened). The `Proceed` path performs no DB writes and does not call - /// this. - /// - /// # Why a threshold here, but no closed-frontier check - /// - /// In the Proceed path no flush ran, so closed batches past the gold - /// frontier (if any) might still be in their natural lifecycle — - /// pending in the mempool, recently included, awaiting safe finality. - /// Cascading them would prematurely abort their progression. - /// - /// The Tip is different: it has no L1 footprint at all (no `w_nonce`, - /// no `safe_input`), so there's no L1 outcome to wait on. Once its - /// first frame has aged into the danger zone, the rule "everything - /// past gold is bad once we're committed to recovery" applies, and in - /// the `RecoverTip` path startup is already committed. - /// - /// # Threshold = danger_threshold, not MAX_WAIT - /// - /// We use `danger_threshold` (= `MAX_WAIT_BLOCKS - margin`) rather than - /// `MAX_WAIT_BLOCKS`. The Tip threshold is the same one that would - /// trigger the recovery cycle had the Tip been a closed batch. If the - /// Tip is past that threshold, the next danger detector tick after - /// resume would re-trip on the Tip's eventual first close + submission - /// anyway (the closed batch would inherit its first frame's safe_block). - /// Cascading now saves the cycle. - /// - /// # Precondition - /// - /// As with [`Storage::recover_post_flush`], the caller must have synced - /// L1 state. (Threshold comparison reads `current_safe_block` from - /// `l1_safe_head`.) - /// - /// Returns the newly-invalidated batch indices (empty if Tip is fresh, - /// `[tip_index]` when the Tip was cascaded). + /// Test-only unguarded primitive; production calls + /// [`Storage::recover_aging_tip_for_recovery`], which transactionally + /// reasserts the exact reducer decision. Design rationale on + /// [`recover_aging_tip_inner`], the shared body. + #[cfg(test)] pub fn recover_aging_tip(&mut self, danger_threshold: u64) -> Result> { self.write(|tx| recover_aging_tip_inner(tx, danger_threshold)) } } +pub(super) fn inspect_recovery_in( + conn: &Connection, + protocol: &ProtocolTiming, + now_ms: u64, +) -> Result { + let danger = check_danger_in(conn, protocol, now_ms)?; + let has_finalized_snapshot = has_finalized_snapshot_in(conn)?; + Ok(RecoveryInspection { + danger, + has_finalized_snapshot, + has_open_tip: has_valid_open_batch(conn)?, + current_safe_block: super::queries::current_safe_block(conn)?, + }) +} + +fn has_finalized_snapshot_in(conn: &Connection) -> Result { + conn.query_row( + "SELECT EXISTS(SELECT 1 FROM finalized_snapshot)", + [], + |row| row.get(0), + ) +} + +fn check_danger_in( + conn: &Connection, + protocol: &ProtocolTiming, + now_ms: u64, +) -> Result { + // The divergence marker outranks everything — including the L1-staleness + // gate. It records an already-confirmed canonical fact. + if let Some((nonce, _)) = canonical_divergence_in(conn)? { + return Ok(DangerStatus::CanonicalDivergence(nonce)); + } + + let safe_block_timestamp = current_safe_block_timestamp(conn)?; + let last_progress = last_safe_progress_ms(conn)?; + if protocol.l1_view_is_stale(safe_block_timestamp, now_ms) { + return Ok(DangerStatus::L1ViewStale); + } + + let danger_threshold = protocol.danger_threshold(); + if let Some(idx) = find_closed_frontier_batch_in_danger(conn, danger_threshold)? { + return Ok(DangerStatus::ClosedBatchInDanger(idx)); + } + if let Some(idx) = find_tip_batch_in_danger(conn, danger_threshold)? { + return Ok(DangerStatus::TipInDanger(idx)); + } + + if protocol.clock_cannot_age_l1_view(safe_block_timestamp, now_ms) { + return Ok(DangerStatus::L1ViewStale); + } + let adjusted_danger_threshold = + match protocol.wall_clock_adjusted_danger_threshold(last_progress, now_ms) { + Ok(adjusted) => adjusted, + Err(_) => return Ok(DangerStatus::L1ViewStale), + }; + if let Some(adjusted) = adjusted_danger_threshold + && let Some(idx) = find_first_batch_in_danger(conn, adjusted)? + { + return Ok(DangerStatus::EstimatedBatchInDanger(idx)); + } + Ok(DangerStatus::Safe) +} + +fn refuse_divergence(danger: DangerStatus) -> std::result::Result<(), RecoveryMutationError> { + if let DangerStatus::CanonicalDivergence(nonce) = danger { + return Err(RecoveryMutationError::CanonicalDivergence { nonce }); + } + Ok(()) +} + // ── Free functions used by both recovery and the batch submitter ────────── -/// See [`Storage::recover_post_flush`] for the design rationale. +/// Cascade the non-gold suffix and open a fresh recovery batch (the shared +/// body behind [`Storage::recover_post_flush_for_recovery`], which the +/// reducer reaches after carrying a Flush witness through a caught-up +/// Sync). Homed here, not on the test wrapper, so rustdoc builds it and a +/// wrapper cleanup cannot delete the design record. +/// +/// # The "everything past gold is doomed" rule +/// +/// At this point the gold frontier is at its maximum extent: every +/// submitted batch has either been accepted (gold) or rejected by the +/// scheduler simulation (Silver-stale, since nonce-mismatch is impossible +/// at the frontier under self-trust), or its tx was killed by a flush +/// no-op (Pending, no `safe_input`). All three non-gold states are doomed: +/// +/// - **Silver-stale:** scheduler skipped it; downstream batches are +/// nonce-poisoned. +/// - **Pending:** the original L1 tx is dead. Re-submission could in +/// principle land fresh, but the *next* recovery cycle's flush would +/// compete with the resub at its new wallet-nonce slot and the bumped +/// no-op typically wins. The system would loop until current staleness +/// crossed `MAX_WAIT_BLOCKS`. Cascading now converges in one cycle. +/// +/// So once we've committed to recovery (the danger detector tripped, the +/// flush ran), the right move is to cascade the entire non-gold suffix +/// and open a fresh recovery batch. +/// +/// Three aftermath shapes: +/// +/// 1. **Everything worked:** all in-flight batches landed fresh and were +/// accepted. Gold extends to the last submitted batch; no first +/// non-gold closed. (See "Tip handling" below for the subtle subcase.) +/// 2. **Mixed:** some landed (stale or poisoned), some replaced. First +/// non-gold closed is either Silver-stale or Pending. Cascade from +/// there; the `batch_index >= N` rule catches the rest of the suffix +/// including the open Tip. +/// 3. **All replaced:** flush no-ops won every race. Gold doesn't +/// advance; first non-gold closed is the very first non-accepted batch. +/// +/// # Tip handling +/// +/// In cases (2)/(3) the cascade catches the Tip via `batch_index >= N`. +/// In case (1), there's no closed pivot — but the Tip can still be in +/// the danger zone: +/// +/// When the lane rotates a batch without a safe-block advance between +/// frames (e.g. immediately after init, when both share the bootstrap +/// `safe_block`), the Tip's `first_frame.safe_block` equals the closed +/// batch's. The closed batch can become gold by inclusion-staleness +/// (`inclusion_block - first_frame < MAX_WAIT`) while the Tip's age, +/// computed against `current_safe_block` after the flush wait, has +/// crossed `danger_threshold`. Pure monotonicity (`S_tip ≥ S_closed`) doesn't +/// rule this out — equality is allowed. +/// +/// So in the no-pivot branch we additionally check the Tip against +/// `danger_threshold` (the same threshold that would have triggered +/// recovery had the Tip been a closed batch). We're already committed +/// to recovery; the Tip is past gold; if it's also in the danger zone, +/// cascade it and open a fresh one. +/// +/// # Atomicity +/// +/// Runs as a single SQLite write transaction. On crash mid-way, the +/// txn rolls back; on commit, the cascade and the recovery batch open +/// land together. Idempotent on re-run because `valid_*` views filter +/// out already-invalidated rows. +/// +/// # Precondition +/// +/// The caller MUST have just synced L1 state via +/// [`Storage::append_safe_inputs`]; the gold frontier in +/// `safe_accepted_batches` must reflect the latest safe head. Otherwise +/// the cascade may invalidate batches that haven't yet had a chance to +/// be processed by the scheduler simulation. +/// +/// Returns the newly-invalidated batch indices (empty if none). fn recover_post_flush_inner(tx: &Transaction<'_>, danger_threshold: u64) -> Result> { // Path 1: any closed batch past gold cascades unconditionally. let pivot = match first_non_gold_closed_batch(tx)? { @@ -339,7 +513,43 @@ fn recover_post_flush_inner(tx: &Transaction<'_>, danger_threshold: u64) -> Resu cascade_and_reopen(tx, pivot) } -/// See [`Storage::recover_aging_tip`] for the design rationale. +/// Cascade the open Tip if its first frame has aged past +/// `danger_threshold` (the shared body behind +/// [`Storage::recover_aging_tip_for_recovery`]). Homed here, not on the +/// test wrapper, so rustdoc builds it and a wrapper cleanup cannot delete +/// the design record. +/// +/// # Why a threshold here, but no closed-frontier check +/// +/// Outside a flush path, closed batches past the gold +/// frontier (if any) might still be in their natural lifecycle — +/// pending in the mempool, recently included, awaiting safe finality. +/// Cascading them would prematurely abort their progression. +/// +/// The Tip is different: it has no L1 footprint at all (no `w_nonce`, +/// no `safe_input`), so there's no L1 outcome to wait on. Once its +/// first frame has aged into the danger zone, the rule "everything +/// past gold is bad once we're committed to recovery" applies, and in +/// the `RecoverTip` path startup is already committed. +/// +/// # Threshold = danger_threshold, not MAX_WAIT +/// +/// We use `danger_threshold` (= `MAX_WAIT_BLOCKS - margin`) rather than +/// `MAX_WAIT_BLOCKS`. The Tip threshold is the same one that would +/// trigger the recovery cycle had the Tip been a closed batch. If the +/// Tip is past that threshold, the next danger detector tick after +/// resume would re-trip on the Tip's eventual first close + submission +/// anyway (the closed batch would inherit its first frame's safe_block). +/// Cascading now saves the cycle. +/// +/// # Precondition +/// +/// As with [`Storage::recover_post_flush`], the caller must have synced +/// L1 state. (Threshold comparison reads `current_safe_block` from +/// `l1_safe_head`.) +/// +/// Returns the newly-invalidated batch indices (empty if Tip is fresh, +/// `[tip_index]` when the Tip was cascaded). fn recover_aging_tip_inner(tx: &Transaction<'_>, danger_threshold: u64) -> Result> { let pivot = find_tip_batch_in_danger(tx, danger_threshold)?; cascade_and_reopen(tx, pivot) @@ -356,8 +566,8 @@ fn recover_aging_tip_inner(tx: &Transaction<'_>, danger_threshold: u64) -> Resul /// Gold-but-unpromoted pendings (batches that landed while the process /// was down) carry lower nonces and *survive*: catch-up resumes from a /// fresher checkpoint, and the rows are cleaned up by the next -/// promotion's `DELETE <= max_nonce`. Scoping is load-bearing (review -/// F9): a blanket clear would arm a promote-wedge crash-loop whenever a +/// promotion's `DELETE <= max_nonce`. Scoping is load-bearing: a +/// blanket clear would arm a promote-wedge crash-loop whenever a /// *valid in-flight* closed batch existed at clear time — its pending /// row would be deleted while the batch stayed valid, and the lane's /// later promotion of its landing would hit the deleted row with no @@ -366,7 +576,11 @@ fn recover_aging_tip_inner(tx: &Transaction<'_>, danger_threshold: u64) -> Resul /// or belongs to a post-recovery batch with a fresh row. In the /// `RecoverTip` path the scope deletes nothing — the Tip never has a /// pending row. Finalized is untouched (L1-confirmed bytes). -/// 3. **Reopen the Tip** the cascade just invalidated (or one a torn crash +/// 3. **Advance `RecoveryGeneration`** exactly once when the cascade +/// invalidated any valid batch. This is the externally visible statement +/// that the current era's soft-history reality changed; composing it here +/// makes generation and invalidation inseparable across crashes. +/// 4. **Reopen the Tip** the cascade just invalidated (or one a torn crash /// left missing), atomically with the cascade. Same mechanism the /// runtime's genesis path uses — see `ingress::open_fresh_tip_in_tx`. fn cascade_and_reopen(tx: &Transaction<'_>, pivot: Option) -> Result> { @@ -379,6 +593,9 @@ fn cascade_and_reopen(tx: &Transaction<'_>, pivot: Option) -> Result Vec::new(), }; + if !invalidated.is_empty() { + advance_recovery_generation_in(tx)?; + } if !invalidated.is_empty() || !has_valid_open_batch(tx)? { open_fresh_tip_in_tx(tx)?; } @@ -484,17 +701,16 @@ fn batch_in_danger(conn: &Connection, batch_index: u64, threshold: u64) -> Resul } /// `frames.safe_block` of the lowest `frame_in_batch` in `batch_index`. -/// Returns 0 if the batch has no frames yet. +/// +/// Every committed valid batch has a first frame. Missing one is an +/// invariant violation and propagates as `QueryReturnedNoRows`. fn first_frame_safe_block_of(conn: &Connection, batch_index: i64) -> Result { - let value: Option = conn - .query_row( - "SELECT safe_block FROM frames \ - WHERE batch_index = ?1 ORDER BY frame_in_batch ASC LIMIT 1", - params![batch_index], - |row| row.get(0), - ) - .optional()?; - Ok(i64_to_u64(value.unwrap_or(0))) + conn.query_row( + "SELECT safe_block FROM frames \ + WHERE batch_index = ?1 ORDER BY frame_in_batch ASC LIMIT 1", + params![batch_index], + |row| row.get::<_, i64>(0).map(i64_to_u64), + ) } /// Cascade-invalidate all valid batches with `batch_index >= from_batch_index`. diff --git a/sequencer/src/storage/recovery_tests.rs b/sequencer/src/storage/recovery_tests.rs index 336bc83a..cc547af8 100644 --- a/sequencer/src/storage/recovery_tests.rs +++ b/sequencer/src/storage/recovery_tests.rs @@ -1,11 +1,281 @@ use super::super::test_helpers::{ SENDER_A, all_ordered_l2_txs, default_protocol_timing, local_batch_payload, - make_stale_batch_payload, seed_closed_batches, seed_safe_inputs_with_batch_nonces, temp_db, + make_stale_batch_payload, seed_closed_batches, seed_safe_inputs_with_batch_nonces, + temp_db_with_default_deployment_identity as temp_db, }; use super::{find_closed_frontier_batch_in_danger, find_first_batch_in_danger}; -use crate::storage::{SafeInputRange, Storage, StoredSafeInput}; +use crate::storage::{ + DirectInputExecution, ExecutedInputCount, SafeInputRange, Storage, StoredSafeInput, +}; use alloy_primitives::Address; use sequencer_core::l2_tx::SequencedL2Tx; +use sequencer_core::protocol::ProtocolTiming; + +mod guarded_phases { + use super::*; + use crate::storage::RecoveryMutationError; + + fn invalidated_count(storage: &Storage) -> u64 { + storage + .conn + .query_row( + "SELECT COUNT(*) FROM batches WHERE invalidated_at_ms IS NOT NULL", + [], + |row| row.get::<_, i64>(0), + ) + .expect("count invalidated batches") as u64 + } + + fn open_tip_count(storage: &Storage) -> u64 { + storage + .conn + .query_row("SELECT COUNT(*) FROM valid_open_batch", [], |row| { + row.get::<_, i64>(0) + }) + .expect("count open Tips") as u64 + } + + fn ensure_tip_fixture( + name: &str, + with_finalized_snapshot: bool, + ) -> ( + crate::storage::test_helpers::TestDb, + Storage, + ProtocolTiming, + ) { + let db = temp_db(name); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let protocol = default_protocol_timing(); + let now_ms = crate::clock::unix_now_ms(); + storage + .append_safe_inputs_with_timestamp( + 0, + now_ms / 1_000, + &[], + SENDER_A, + &protocol, + crate::storage::FrontierMode::Populate, + ) + .expect("seed fresh safe head"); + if with_finalized_snapshot { + let prefix = db._dir.path().join("finalized"); + storage + .insert_finalized_dump(&prefix, 0, 0) + .expect("seed finalized snapshot"); + } + (db, storage, protocol) + } + + #[test] + fn ensure_tip_guard_opens_one_tip_only_from_clean_facts() { + let (_db, mut storage, protocol) = ensure_tip_fixture("guarded-ensure-tip", true); + assert_eq!(open_tip_count(&storage), 0); + + storage + .ensure_open_tip_for_recovery(&protocol, crate::clock::unix_now_ms()) + .expect("clean no-Tip facts authorize one Tip"); + assert_eq!(open_tip_count(&storage), 1); + } + + #[test] + fn ensure_tip_guard_refuses_missing_snapshot_without_mutation() { + let (_db, mut storage, protocol) = + ensure_tip_fixture("guarded-ensure-tip-no-snapshot", false); + storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("a stale phase opened the Tip"); + + let error = storage + .ensure_open_tip_for_recovery(&protocol, crate::clock::unix_now_ms()) + .expect_err("missing finalized state must outrank the stale Tip decision"); + assert!(matches!( + error, + RecoveryMutationError::MissingFinalizedSnapshot + )); + assert_eq!(open_tip_count(&storage), 1, "the existing Tip is untouched"); + } + + #[test] + fn ensure_tip_guard_refuses_divergence_without_mutation() { + let (_db, mut storage, protocol) = + ensure_tip_fixture("guarded-ensure-tip-divergence", true); + crate::storage::test_helpers::record_canonical_divergence(&mut storage, 7, 0); + + let error = storage + .ensure_open_tip_for_recovery(&protocol, crate::clock::unix_now_ms()) + .expect_err("divergence must outrank Tip creation"); + assert!(matches!( + error, + RecoveryMutationError::CanonicalDivergence { nonce: 7 } + )); + assert_eq!(open_tip_count(&storage), 0); + } + + #[test] + fn ensure_tip_guard_rejects_an_already_open_tip_without_duplication() { + let (_db, mut storage, protocol) = ensure_tip_fixture("guarded-ensure-tip-existing", true); + storage + .initialize_open_state(0, SafeInputRange::empty_at(0)) + .expect("a competing phase opened the Tip"); + + let error = storage + .ensure_open_tip_for_recovery(&protocol, crate::clock::unix_now_ms()) + .expect_err("stale no-Tip decision must be rejected"); + assert!(matches!( + error, + RecoveryMutationError::StaleDecision { + expected: crate::storage::DangerStatus::Safe, + actual: crate::storage::DangerStatus::Safe, + } + )); + assert_eq!(open_tip_count(&storage), 1); + } + + fn cascade_fixture(name: &str) -> (crate::storage::test_helpers::TestDb, Storage) { + let db = temp_db(name); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let protocol = default_protocol_timing(); + let mut head = storage + .initialize_open_state(10, SafeInputRange::empty_at(0)) + .expect("initialize Tip"); + storage + .close_frame_and_batch(&mut head, 10) + .expect("close cascade candidate"); + storage + .append_safe_inputs_with_timestamp( + 1_200, + 1_000, + &[], + SENDER_A, + &protocol, + crate::storage::FrontierMode::Populate, + ) + .expect("advance safe head"); + let prefix = db._dir.path().join("finalized"); + storage + .insert_finalized_dump(&prefix, 0, 0) + .expect("seed finalized snapshot"); + (db, storage) + } + + #[test] + fn cascade_guard_ranks_divergence_before_f2_and_mutates_nothing() { + let (_db, mut storage) = cascade_fixture("guarded-cascade-divergence"); + let protocol = default_protocol_timing(); + crate::storage::test_helpers::record_canonical_divergence(&mut storage, 0, 0); + + let error = storage + .recover_post_flush_for_recovery(1_201, &protocol, crate::clock::unix_now_ms()) + .expect_err("divergence must refuse before lag"); + assert!(matches!( + error, + RecoveryMutationError::CanonicalDivergence { nonce: 0 } + )); + assert_eq!(invalidated_count(&storage), 0); + } + + #[test] + fn cascade_guard_rejects_lagging_sync_without_tree_mutation() { + let (_db, mut storage) = cascade_fixture("guarded-cascade-f2"); + let protocol = default_protocol_timing(); + + let error = storage + .recover_post_flush_for_recovery(1_201, &protocol, crate::clock::unix_now_ms()) + .expect_err("lagging post-flush view must retry"); + assert!(matches!( + error, + RecoveryMutationError::ResyncBehindFlushView { + resynced_safe_block: 1_200, + flush_observed_safe_block: 1_201, + } + )); + assert_eq!(invalidated_count(&storage), 0); + } + + #[test] + fn cascade_guard_ranks_missing_snapshot_before_f2_without_mutation() { + let (_db, mut storage) = cascade_fixture("guarded-cascade-no-snapshot"); + let protocol = default_protocol_timing(); + storage + .conn + .execute("DELETE FROM finalized_snapshot", []) + .expect("remove finalized snapshot fact"); + + let error = storage + .recover_post_flush_for_recovery(1_201, &protocol, crate::clock::unix_now_ms()) + .expect_err("missing finalized state must outrank a lagging view"); + assert!(matches!( + error, + RecoveryMutationError::MissingFinalizedSnapshot + )); + assert_eq!(invalidated_count(&storage), 0); + } + + #[test] + fn tip_guard_reasserts_the_exact_reducer_decision() { + let db = temp_db("guarded-tip-decision"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let protocol = default_protocol_timing(); + storage + .initialize_open_state(10, SafeInputRange::empty_at(0)) + .expect("initialize Tip"); + storage + .append_safe_inputs_with_timestamp( + 1_200, + 1_000, + &[], + SENDER_A, + &protocol, + crate::storage::FrontierMode::Populate, + ) + .expect("advance safe head"); + let prefix = db._dir.path().join("finalized"); + storage + .insert_finalized_dump(&prefix, 0, 0) + .expect("seed finalized snapshot"); + + let error = storage + .recover_aging_tip_for_recovery(1, &protocol, 1_000_000) + .expect_err("a stale expected Tip must not mutate"); + assert!(matches!( + error, + RecoveryMutationError::StaleDecision { + expected: crate::storage::DangerStatus::TipInDanger(1), + actual: crate::storage::DangerStatus::TipInDanger(0), + } + )); + assert_eq!(invalidated_count(&storage), 0); + } + + #[test] + fn tip_guard_refuses_missing_snapshot_without_tree_mutation() { + let db = temp_db("guarded-tip-no-snapshot"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let protocol = default_protocol_timing(); + storage + .initialize_open_state(10, SafeInputRange::empty_at(0)) + .expect("initialize Tip"); + storage + .append_safe_inputs_with_timestamp( + 1_200, + 1_000, + &[], + SENDER_A, + &protocol, + crate::storage::FrontierMode::Populate, + ) + .expect("advance safe head"); + + let error = storage + .recover_aging_tip_for_recovery(0, &protocol, 1_000_000) + .expect_err("missing finalized state must refuse Tip recovery"); + assert!(matches!( + error, + RecoveryMutationError::MissingFinalizedSnapshot + )); + assert_eq!(invalidated_count(&storage), 0); + } +} mod invalid_batches { use super::*; @@ -214,6 +484,8 @@ mod recover_post_flush { fn is_idempotent() { let db = temp_db("detect-idempotent"); let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let initial_history = storage.history_state().expect("initial history"); + assert_eq!(initial_history.version.recovery_generation.get(), 0); let mut head = storage .initialize_open_state(10, SafeInputRange::empty_at(0)) @@ -237,9 +509,38 @@ mod recover_post_flush { .expect("append safe input"); let first = storage.recover_post_flush(1200).expect("first detect"); assert_eq!(first, vec![0, 1]); + assert_eq!( + storage + .history_state() + .expect("history after recovery") + .version + .recovery_generation + .get(), + 1, + "one invalidating recovery advances exactly once" + ); + assert_eq!( + storage + .history_state() + .expect("history after recovery") + .version + .era_id, + initial_history.version.era_id, + "standard recovery stays within the same era" + ); let second = storage.recover_post_flush(1200).expect("second detect"); assert!(second.is_empty()); + assert_eq!( + storage + .history_state() + .expect("history after no-op") + .version + .recovery_generation + .get(), + 1, + "a recovery no-op must not invent a new history reality" + ); } #[test] @@ -278,6 +579,15 @@ mod recover_post_flush { // batch 2 opened with nonce reused (= 0). let first = storage.recover_post_flush(1200).expect("gen1 recovery"); assert_eq!(first, vec![0, 1]); + assert_eq!( + storage + .history_state() + .expect("generation one") + .version + .recovery_generation + .get(), + 1 + ); // Submitter posts the recovery batch; it lands fresh on L1. let mut head = storage.open_state().expect("load").unwrap(); @@ -362,6 +672,15 @@ mod recover_post_flush { vec![2, 3], "stale reused nonce in gen2 must still be detected" ); + assert_eq!( + storage + .history_state() + .expect("generation two") + .version + .recovery_generation + .get(), + 2 + ); } #[test] @@ -496,7 +815,7 @@ mod tip_staleness { // - boundary at threshold: invalidated // - boundary just below threshold: not invalidated // - // The remaining tests cover the FlushAndCascade path's combined + // The remaining tests cover the guarded post-flush Cascade phase's combined // closed+open behavior (`recover_post_flush`'s `batch_index >= N` // cascade rule catches the Tip too). @@ -526,6 +845,16 @@ mod tip_staleness { vec![0], "open batch 0 should be invalidated by current staleness" ); + assert_eq!( + storage + .history_state() + .expect("history after Tip recovery") + .version + .recovery_generation + .get(), + 1, + "the direct RecoverTip path advances the shared generation boundary" + ); // A fresh recovery batch must be opened at batch_index=1. let head = storage.open_state().expect("load").expect("head"); @@ -655,6 +984,16 @@ mod tip_staleness { .recover_post_flush(1200) .expect("recover from torn state"); assert!(invalidated.is_empty(), "no new invalidations"); + assert_eq!( + storage + .history_state() + .expect("history after structural repair") + .version + .recovery_generation + .get(), + 0, + "opening a missing Tip without invalidating history is not a recovery generation" + ); let head = storage.open_state().expect("load open state"); assert!(head.is_some(), "recovery should have opened a fresh batch"); @@ -732,6 +1071,121 @@ mod tip_staleness { ); } + #[test] + fn rolls_back_generation_and_invalidation_when_tip_reopen_aborts() { + let db = temp_db("detect-tip-reopen-abort"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + + let mut head = storage + .initialize_open_state(10, SafeInputRange::empty_at(0)) + .expect("initialize"); + storage + .append_safe_inputs( + 10, + &[StoredSafeInput { + sender: Address::ZERO, + payload: vec![0xd1], + block_number: 10, + }], + SENDER_A, + &default_protocol_timing(), + ) + .expect("append mapped direct"); + storage + .close_frame_only_with_executions( + &mut head, + 10, + SafeInputRange::new(0, 1), + &[DirectInputExecution { + safe_input_index: 0, + executed_input_offset: ExecutedInputCount::ZERO, + }], + ) + .expect("attribute mapped direct"); + storage + .close_frame_and_batch(&mut head, 10) + .expect("close batch 0"); + assert_eq!( + storage + .next_executed_input_count() + .expect("pre-recovery history head"), + ExecutedInputCount::new(1) + ); + storage + .append_safe_inputs(1500, &[], SENDER_A, &default_protocol_timing()) + .expect("advance safe head past staleness"); + + storage + .conn + .execute_batch( + "CREATE TRIGGER fail_recovery_tip_insert + BEFORE INSERT ON batches + BEGIN + SELECT RAISE(ABORT, 'injected Tip reopen failure'); + END;", + ) + .expect("install failure trigger"); + + let err = storage + .recover_post_flush(1200) + .expect_err("Tip reopen failure must abort the whole recovery transaction"); + assert!( + err.to_string().contains("injected Tip reopen failure"), + "unexpected error: {err:?}" + ); + assert_eq!( + storage + .history_state() + .expect("history after rollback") + .version + .recovery_generation + .get(), + 0, + "the generation bump must roll back with the failed Tip reopen" + ); + let invalidated_count: i64 = storage + .conn + .query_row( + "SELECT COUNT(*) FROM batches WHERE invalidated_at_ms IS NOT NULL", + [], + |row| row.get(0), + ) + .expect("count invalidated"); + assert_eq!( + invalidated_count, 0, + "the cascade must roll back with the generation bump" + ); + let mappings: Vec = storage + .conn + .prepare( + "SELECT executed_input_offset FROM executed_inputs ORDER BY executed_input_offset", + ) + .expect("prepare mapping query") + .query_map([], |row| row.get(0)) + .expect("query mappings") + .collect::>() + .expect("collect mappings"); + assert_eq!( + mappings, + vec![0], + "trigger-deleted mappings must roll back with the failed recovery" + ); + assert_eq!( + storage + .next_executed_input_count() + .expect("history head after rollback"), + ExecutedInputCount::new(1), + "the canonical history head must roll back with its mapping" + ); + let open_batch_index: i64 = storage + .conn + .query_row("SELECT batch_index FROM valid_open_batch", [], |row| { + row.get(0) + }) + .expect("query original Tip"); + assert_eq!(open_batch_index, 1); + } + #[test] fn recovery_redrains_direct_inputs_and_replay_sees_them_once() { let db = temp_db("recovery-redrain-e2e"); @@ -1293,7 +1747,7 @@ mod check_any_unresolved { // When BOTH the closed frontier batch and the open Tip are aged past the // threshold, find_first_batch_in_danger must return the CLOSED frontier: // cascading from it covers the Tip (batch_index >= pivot), and the scoped - // pending-snapshot clear keys on the pivot's nonce (F9). Both existing + // pending-snapshot clear keys on the pivot's nonce. Both existing // find_*_in_danger tests use open-batch-only scenarios; the closed-over-Tip // preference (the helper's whole point) was unasserted. let db = temp_db("danger-prefers-closed-frontier"); @@ -1933,7 +2387,7 @@ mod schema_invariants { #[test] fn schema_rejects_payload_hash_rewrite() { - // payload_hash is the content-identity anchor for the R2 canonical- + // payload_hash is the content-identity anchor for the canonical- // divergence check; a rewrite would let a foreign/zombie L1 landing // false-match a local batch. Write-once once stamped at seal. let db = temp_db("schema-payload-hash-write-once"); @@ -2128,7 +2582,7 @@ mod schema_invariants { // collide with the EIP-712 domain's unspecified-chain sentinel // and break signature recovery; the CHECK refuses to persist it // in the first place. - let db = temp_db("schema-deployment-chain-id-zero"); + let db = crate::storage::test_helpers::temp_db("schema-deployment-chain-id-zero"); let storage = Storage::open(db.path.as_str()).expect("open storage"); let address = vec![0u8; 20]; let err = storage.conn.execute( @@ -2167,6 +2621,84 @@ mod schema_invariants { ); } + #[test] + fn divergence_marker_freezes_batch_tree_promotions_and_pending_clears() { + // I15 structural enforcement: with the canonical-divergence marker + // present, the trg_*_frozen_on_divergence family must RAISE on batch + // inserts/updates, promotion inserts, and pending-snapshot deletes — + // even for a caller that bypassed every typed Rust refusal. + let db = temp_db("schema-divergence-freeze"); + let mut storage = Storage::open(db.path.as_str()).expect("open storage"); + let mut head = storage + .initialize_open_state(10, SafeInputRange::empty_at(0)) + .expect("initialize open state"); + storage + .close_frame_and_batch(&mut head, 10) + .expect("close batch 0 before freezing"); + storage + .insert_pending_dump(std::path::Path::new("/tmp/pending-fixture"), 1, 0) + .expect("insert a pending dump row to attempt deleting"); + crate::storage::test_helpers::record_canonical_divergence(&mut storage, 0, 0); + + let frozen = |err: rusqlite::Error| { + let text = format!("{err}"); + assert!( + text.contains("frozen") && text.contains("divergence"), + "expected a divergence-freeze RAISE, got: {text}" + ); + }; + + frozen( + storage + .conn + .execute( + "INSERT INTO batches (batch_index, parent_batch_index, nonce, created_at_ms) \ + VALUES (99, NULL, 0, 0)", + [], + ) + .expect_err("batch INSERT must be frozen"), + ); + frozen( + storage + .conn + .execute( + "UPDATE batches SET invalidated_at_ms = 1 WHERE invalidated_at_ms IS NULL", + [], + ) + .expect_err("batch UPDATE (cascade shape) must be frozen"), + ); + frozen( + storage + .conn + .execute( + "INSERT OR REPLACE INTO finalized_snapshot \ + (singleton_id, dump_id, inclusion_block, l2_tx_index) \ + VALUES (0, 999, 1, 0)", + [], + ) + .expect_err("promotion INSERT must be frozen"), + ); + frozen( + storage + .conn + .execute("DELETE FROM pending_snapshots", []) + .expect_err("pending-snapshot DELETE must be frozen"), + ); + + // Reads stay usable on a frozen DB: the danger check must still + // classify the divergence (the boot-refusal path depends on it). + let protocol = ProtocolTiming { + max_wait_blocks: 1200, + preemptive_margin_blocks: 75, + l1_read_stale_after_blocks: 900, + seconds_per_block: 12, + }; + let status = storage + .check_danger(&protocol, 1_000_000) + .expect("check danger on a frozen DB"); + assert_eq!(status, crate::storage::DangerStatus::CanonicalDivergence(0)); + } + #[test] fn schema_rejects_safe_input_with_negative_block_timestamp() { let db = temp_db("schema-safe-input-neg-block-timestamp"); @@ -2636,7 +3168,7 @@ mod recovery_clears_pending_snapshots { ); } - /// Review F9 regression: the cascade's pending clear is scoped to + /// Regression test: the cascade's pending clear is scoped to /// `nonce >= pivot.nonce`. A gold-but-unpromoted pending (its batch /// landed accepted while the process was down, the lane never /// promoted it) sits *below* the pivot and must survive — deleting diff --git a/sequencer/src/storage/safe_accepted_batches.rs b/sequencer/src/storage/safe_accepted_batches.rs index d62ef08c..cb4f9448 100644 --- a/sequencer/src/storage/safe_accepted_batches.rs +++ b/sequencer/src/storage/safe_accepted_batches.rs @@ -8,9 +8,11 @@ //! acceptance rules (see [`sequencer_core::protocol::ProtocolTiming`]). //! //! Maintenance contract: the view is advanced atomically with each -//! [`super::Storage::append_safe_inputs`] write, so any reader that sees -//! `l1_safe_head` at block B also sees every acceptance decision up to B. No -//! caller should populate this view directly. +//! [`super::Storage::append_safe_inputs`] write while no divergence exists. A +//! foreign/mismatched accepted landing atomically commits the terminal marker +//! and freezes this projection; later safe-head advances remain paired with +//! that marker rather than further acceptance rows. No caller should populate +//! this view directly. //! //! Readers: //! - batch submitter frontier / danger reads (`submitter_frontier`, @@ -63,7 +65,7 @@ pub(super) fn query_latest_safe_accepted_batch( /// valid path (`trg_enforce_nonce_contiguity`). pub(super) fn frontier_nonce(conn: &Connection) -> Result { match query_latest_safe_accepted_batch(conn)? { - Some(row) => Ok(i64_to_u64(row.nonce).saturating_add(1)), + Some(row) => Ok(next_expected_nonce(i64_to_u64(row.nonce))), // Empty accepted table ⇒ the frontier sits at the batch-tree anchor: 0 // for a genesis deployment, N' for a cockroach-recovered one. Reading // the anchor (not a hard-coded 0) keeps this in step with @@ -79,9 +81,22 @@ pub(super) fn frontier_nonce(conn: &Connection) -> Result { } } +fn next_expected_nonce(nonce: u64) -> u64 { + nonce + .checked_add(1) + .expect("accepted batch nonce overflow: contract-impossible") +} + /// Simulate the scheduler's acceptance logic over new safe inputs and append /// matches to `safe_accepted_batches`. /// +/// The content-identity check (I9/I15) is complete for this mirrored +/// predicate: every at/above-anchor accepted landing is a byte-identical +/// local match, foreign, or mismatched. It is not +/// an independent oracle for the canonical scheduler, application state, or +/// collapsed checkpoint history; foreign/mismatch requires manual cockroach +/// recovery. +/// /// Paginates through `safe_inputs` rows newer than the latest accepted row, /// pre-filtered at SQL to `batch_submitter` as the sender. For each row, /// delegates to [`ProtocolTiming::scheduler_accepts`] with the @@ -108,7 +123,7 @@ pub(super) fn frontier_nonce(conn: &Connection) -> Result { /// nonce-advance fold in `protocol::advance_expected_batch_nonce`. /// This loop deliberately keeps its own inline `expected` advance rather than /// reusing that fold: the advance is interleaved with two storage-only side -/// effects that cannot move below the protocol layer — the R2 content-identity +/// effects that cannot move below the protocol layer — the content-identity /// check ([`content_identity_violation`]) and the `canonical_divergence` freeze. /// Sharing a fold here would force a callback contract for those (a refactor, /// not the no-behavior-change library move). @@ -122,7 +137,7 @@ pub(super) fn populate_safe_accepted_batches( FROM safe_inputs \ WHERE sender = ?1 AND safe_input_index > ?2 \ ORDER BY safe_input_index ASC LIMIT ?3"; - const INSERT_SQL: &str = "INSERT OR IGNORE INTO safe_accepted_batches \ + const INSERT_SQL: &str = "INSERT INTO safe_accepted_batches \ (safe_input_index, nonce, first_frame_safe_block, inclusion_block) \ VALUES (?1, ?2, ?3, ?4)"; @@ -131,7 +146,7 @@ pub(super) fn populate_safe_accepted_batches( // and advancing it (or promoting on it) would compound the divergence. // `check_danger` reports `CanonicalDivergence` ahead of every other arm, // so the detector exits / startup refuses; the remedy is cockroach - // recovery, never standard recovery (review R2). + // recovery, never standard recovery. if canonical_divergence_in(conn)?.is_some() { return Ok(()); } @@ -152,7 +167,7 @@ pub(super) fn populate_safe_accepted_batches( .map(|row| row.safe_input_index) .unwrap_or(-1); let mut expected = latest_accepted - .map(|row| i64_to_u64(row.nonce).saturating_add(1)) + .map(|row| next_expected_nonce(i64_to_u64(row.nonce))) .unwrap_or(anchor); loop { @@ -187,7 +202,7 @@ pub(super) fn populate_safe_accepted_batches( continue; }; - // Content-identity check (review R2), gated on full acceptance — + // Content-identity check, gated on full acceptance — // exactly here, where the simulated scheduler accepted the // landing. Rejected/stale/undecodable copies are scheduler // no-ops; their content is irrelevant by construction. @@ -203,17 +218,22 @@ pub(super) fn populate_safe_accepted_batches( ); // Stop scanning; the marker (committed with this sync) // freezes the frontier and routes every subsequent boot and - // detector tick to refusal. + // detector tick to refusal. The accepted lane-reconciliation + // cutover will also make this the lane's next slow-turn + // terminal result. return Ok(()); } - insert_stmt.execute(params![ + let changed = insert_stmt.execute(params![ u64_to_i64(accepted.safe_input_index), u64_to_i64(accepted.nonce), u64_to_i64(accepted.first_frame_safe_block), u64_to_i64(accepted.inclusion_block), ])?; - expected = expected.saturating_add(1); + if changed != 1 { + return Err(rusqlite::Error::StatementChangedRows(changed)); + } + expected = next_expected_nonce(expected); } if page_len < PAGE_SIZE { @@ -245,8 +265,9 @@ impl DivergenceKind { } } -/// R2 check proper: compare a fully-accepted landing against our valid -/// closed batch at the same nonce. `None` = identical (the normal case). +/// Content-identity check proper: compare a fully-accepted landing against +/// our valid closed batch at the same nonce. `None` = identical (the normal +/// case). /// /// Why content (not identity) suffices: accepted content-equal copies are /// effect-equal — which physical tx landed carries no semantic weight. The @@ -331,3 +352,98 @@ fn record_canonical_divergence_in( )?; Ok(()) } + +#[cfg(test)] +mod tests { + use super::*; + use crate::storage::{Storage, test_helpers::temp_db}; + + fn insert_safe_input_zero(storage: &Storage) { + storage + .conn + .execute( + "INSERT INTO safe_inputs \ + (safe_input_index, sender, payload, block_number, block_timestamp, transaction_hash) \ + VALUES (0, ?1, X'', 0, 0, ?2)", + params![Address::ZERO.as_slice(), [0_u8; 32].as_slice()], + ) + .expect("insert parent safe input"); + } + + #[test] + fn safe_accepted_batch_nonce_constraint_rejects_negative_values() { + let db = temp_db("safe-accepted-negative-nonce"); + let storage = Storage::open(db.path.as_str()).expect("open storage"); + insert_safe_input_zero(&storage); + + let err = storage + .conn + .execute( + "INSERT INTO safe_accepted_batches \ + (safe_input_index, nonce, first_frame_safe_block, inclusion_block) \ + VALUES (0, -1, 0, 0)", + [], + ) + .expect_err("negative accepted-batch nonce must violate the schema"); + + assert!( + matches!( + err, + rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code: rusqlite::ErrorCode::ConstraintViolation, + .. + }, + _ + ) + ), + "unexpected error: {err}" + ); + } + + #[test] + fn accepted_batch_insert_conflicts_fail_loud() { + let db = temp_db("safe-accepted-duplicate"); + let storage = Storage::open(db.path.as_str()).expect("open storage"); + insert_safe_input_zero(&storage); + storage + .conn + .execute( + "INSERT INTO safe_accepted_batches \ + (safe_input_index, nonce, first_frame_safe_block, inclusion_block) \ + VALUES (0, 0, 0, 0)", + [], + ) + .expect("insert accepted row"); + + let err = storage + .conn + .execute( + "INSERT INTO safe_accepted_batches \ + (safe_input_index, nonce, first_frame_safe_block, inclusion_block) \ + VALUES (0, 0, 0, 0)", + [], + ) + .expect_err("duplicate accepted row must not be ignored"); + + assert!( + matches!( + err, + rusqlite::Error::SqliteFailure( + rusqlite::ffi::Error { + code: rusqlite::ErrorCode::ConstraintViolation, + .. + }, + _ + ) + ), + "unexpected error: {err}" + ); + } + + #[test] + #[should_panic(expected = "accepted batch nonce overflow: contract-impossible")] + fn accepted_batch_nonce_advance_fails_loud_on_overflow() { + let _ = next_expected_nonce(u64::MAX); + } +} diff --git a/sequencer/src/storage/snapshot_dumps.rs b/sequencer/src/storage/snapshot_dumps.rs index a56cd4e5..de5e6d97 100644 --- a/sequencer/src/storage/snapshot_dumps.rs +++ b/sequencer/src/storage/snapshot_dumps.rs @@ -14,11 +14,14 @@ //! drain), GC after each promotion. use std::path::{Path, PathBuf}; +use std::sync::Arc; use rusqlite::{OptionalExtension, Result, Transaction, params}; -use super::Storage; use super::convert::{i64_to_u64, u64_to_i64}; +use super::history::{bind_history_base_in, next_executed_input_count_in}; +use super::{Storage, is_persistent_storage_error, is_persistent_storage_open_error}; +use sequencer_core::history::ExecutedInputCount; /// A row in `dumps`: SQLite primary key plus the on-disk directory. #[derive(Debug, Clone, PartialEq, Eq)] @@ -33,6 +36,7 @@ pub struct PendingDump { pub nonce: u64, pub dump: DumpRow, pub l2_tx_index: u64, + pub executed_input_count: ExecutedInputCount, } /// The singleton `finalized_snapshot` row joined with the underlying @@ -42,15 +46,18 @@ pub struct FinalizedDump { pub dump: DumpRow, pub inclusion_block: u64, pub l2_tx_index: u64, + pub executed_input_count: ExecutedInputCount, } -/// How a [`LeaseGuard`] runs its (blocking) release on drop. Injected by the +/// How a [`LeaseGuard`] schedules its blocking release on drop. Injected by the /// caller so storage stays runtime-agnostic: the egress HTTP layer passes a -/// scheduler that offloads to `tokio::spawn_blocking` (so a release triggered -/// by client disconnect doesn't stall an async worker); sync callers and tests -/// pass one that runs it inline. Same fn-pointer decoupling as -/// `egress::api::snapshot`'s `state_file_in_dump`. -pub type ReleaseScheduler = fn(release: Box); +/// supervised queue, while sync callers and tests pass one that runs inline. +/// The owned callback lets each guard retain the queue's producer token until +/// its `Drop` has submitted the release. A separate required reporter carries +/// only the persistent-failure signal back to the runtime, keeping storage +/// independent of runtime shutdown types. +pub type ReleaseScheduler = Arc) + Send + Sync + 'static>; +pub type PersistentReleaseFailureReporter = Arc; /// An armed lease release, inseparable from the lease it holds. Handed out /// bundled inside a [`LeasedDump`]: while it lives, `lease_count > 0` keeps GC @@ -63,25 +70,52 @@ pub struct LeaseGuard { path: String, dump_id: i64, schedule: ReleaseScheduler, + report_persistent_failure: PersistentReleaseFailureReporter, } impl Drop for LeaseGuard { fn drop(&mut self) { let path = std::mem::take(&mut self.path); let dump_id = self.dump_id; + let report_persistent_failure = self.report_persistent_failure.clone(); (self.schedule)(Box::new(move || match Storage::open_writer(&path) { Ok(mut storage) => { if let Err(err) = storage.release_dump_lease(dump_id) { + // Log before reporting: the reporter blocks on the + // externalization gate, and the operator must see the + // actual error even if publication stalls. + if is_persistent_storage_error(&err) { + tracing::error!( + error = %err, dump_id, + "snapshot lease release failed persistently", + ); + report_persistent_failure(&format!( + "snapshot lease release for dump {dump_id} failed persistently: {err}" + )); + } else { + tracing::warn!( + error = %err, dump_id, + "snapshot lease release failed; will be reset at next startup", + ); + } + } + } + Err(err) => { + if is_persistent_storage_open_error(&err) { + tracing::error!( + error = %err, dump_id, + "snapshot lease release: writer open failed persistently", + ); + report_persistent_failure(&format!( + "snapshot lease release for dump {dump_id}: writer open failed persistently: {err}" + )); + } else { tracing::warn!( error = %err, dump_id, - "snapshot lease release failed; will be reset at next startup", + "snapshot lease release: open failed; will be reset at next startup", ); } } - Err(err) => tracing::warn!( - error = %err, dump_id, - "snapshot lease release: open failed; will be reset at next startup", - ), })); } } @@ -95,6 +129,7 @@ impl Drop for LeaseGuard { pub struct LeasedDump { pub prefix: PathBuf, pub l2_tx_index: u64, + pub executed_input_count: ExecutedInputCount, pub inclusion_block: Option, pub guard: LeaseGuard, } @@ -108,13 +143,19 @@ impl Storage { /// exists in `dumps`; this is intentional — the caller is expected /// to pass fresh, unique prefixes per call, and reuse is a bug /// worth surfacing loudly. - pub fn insert_pending_dump( + /// Test seed only: production stages pending rows exclusively through + /// the lane's atomic `close_batch_with_snapshot` path. + #[cfg(test)] + pub(crate) fn insert_pending_dump( &mut self, prefix: &Path, nonce: u64, l2_tx_index: u64, ) -> Result { - self.write(|tx| insert_pending_dump_in(tx, prefix, nonce, l2_tx_index)) + self.write(|tx| { + let executed_input_count = next_executed_input_count_in(tx)?; + insert_pending_dump_in(tx, prefix, nonce, l2_tx_index, executed_input_count) + }) } /// Atomically promote the pending dump for `max_nonce` into the @@ -133,10 +174,11 @@ impl Storage { /// *supersede* an existing finalized row, which `insert_finalized_dump` /// can't). **Production does not call this**: the lane promotes via /// `promote_finalized_in` folded into the safe-frontier-advance - /// transaction ([`Storage::close_frame_only_promoting`]), so the promotion - /// commits atomically with the drain it derives from — a separate promotion - /// could commit ahead of the drain and wedge a restart on a deleted pending - /// row. + /// transaction + /// ([`Storage::close_frame_only_promoting_with_executions`]), so promotion, + /// drain, and canonical attribution commit atomically. A separate + /// promotion could commit ahead of the drain and wedge a restart on a + /// deleted pending row. pub fn promote_finalized(&mut self, max_nonce: u64, inclusion_block: u64) -> Result<()> { self.write(|tx| promote_finalized_in(tx, max_nonce, inclusion_block)) } @@ -235,11 +277,11 @@ impl Storage { } /// The snapshot to resume or serve from: the latest pending dump, else the - /// finalized snapshot. Returns its `(dump row, l2_tx_index)`, or `None` if - /// neither exists. This is catch-up's resume checkpoint; the leasing variant + /// finalized snapshot. Returns its `(dump row, l2_tx_index, + /// executed_input_count)`, or `None` if neither exists. This is catch-up's resume checkpoint; the leasing variant /// [`Storage::acquire_latest_snapshot_lease`] shares the same "pending else /// finalized" selection via `latest_snapshot_in`. - pub fn latest_snapshot(&mut self) -> Result> { + pub fn latest_snapshot(&mut self) -> Result> { self.read(latest_snapshot_in) } @@ -248,29 +290,47 @@ impl Storage { /// handler reads the row, a promotion + GC delete the dump, and the open /// then fails: the lease is held from the moment of the read. `None` if no /// finalized snapshot exists. `schedule` controls where the (blocking) - /// release runs on drop — see [`ReleaseScheduler`]. + /// release runs on drop — see [`ReleaseScheduler`]. The reporter is + /// required — an unreported persistent release failure must be impossible; + /// it is called only for persistent row/schema failures, never + /// BUSY/I/O. Tests pass a no-op closure. pub fn acquire_finalized_lease( &mut self, schedule: ReleaseScheduler, + report_persistent_failure: PersistentReleaseFailureReporter, ) -> Result> { let path = self.path.clone(); - self.write(|tx| { + let acquired = self.write(|tx| { let Some(f) = finalized_dump_in(tx)? else { return Ok(None); }; let dump_id = f.dump.id; acquire_dump_lease_in(tx, dump_id)?; - Ok(Some(LeasedDump { - prefix: f.dump.prefix, - l2_tx_index: f.l2_tx_index, - inclusion_block: Some(f.inclusion_block), + Ok(Some(( + f.dump, + f.l2_tx_index, + f.executed_input_count, + Some(f.inclusion_block), + ))) + })?; + + Ok(acquired.map( + |(dump, l2_tx_index, executed_input_count, inclusion_block)| LeasedDump { + prefix: dump.prefix, + l2_tx_index, + executed_input_count, + inclusion_block, + // Arm the release only after `Storage::write` has committed the + // increment. A failed COMMIT rolls back the lease and must not + // schedule a decrement for a lease that never existed. guard: LeaseGuard { path, - dump_id, + dump_id: dump.id, schedule, + report_persistent_failure, }, - })) - }) + }, + )) } /// Atomically read the snapshot to serve (latest pending, else finalized) @@ -280,25 +340,34 @@ impl Storage { pub fn acquire_latest_snapshot_lease( &mut self, schedule: ReleaseScheduler, + report_persistent_failure: PersistentReleaseFailureReporter, ) -> Result> { let path = self.path.clone(); - self.write(|tx| { - let Some((dump, l2_tx_index)) = latest_snapshot_in(tx)? else { + let acquired = self.write(|tx| { + let Some((dump, l2_tx_index, executed_input_count)) = latest_snapshot_in(tx)? else { return Ok(None); }; let dump_id = dump.id; acquire_dump_lease_in(tx, dump_id)?; - Ok(Some(LeasedDump { + Ok(Some((dump, l2_tx_index, executed_input_count))) + })?; + + Ok( + acquired.map(|(dump, l2_tx_index, executed_input_count)| LeasedDump { prefix: dump.prefix, l2_tx_index, + executed_input_count, inclusion_block: None, + // See `acquire_finalized_lease`: the guard owns a release only + // after the matching increment is durable. guard: LeaseGuard { path, - dump_id, + dump_id: dump.id, schedule, + report_persistent_failure, }, - })) - }) + }), + ) } /// Return every row in `dumps`. Used at startup to reconcile @@ -311,7 +380,7 @@ impl Storage { /// Delete every row from `pending_snapshots`. Test-only convenience wrapper /// for the *unscoped* clear: production danger-zone recovery instead composes - /// the pivot-scoped `clear_pending_dumps_from_nonce_in` (F9) into the same + /// the pivot-scoped `clear_pending_dumps_from_nonce_in` into the same /// transaction as the cascade invalidation (see `storage/recovery.rs`), so /// only the cascade-doomed batches' pending rows are cleared, atomically with /// them. @@ -324,29 +393,51 @@ impl Storage { /// transaction. Used at first startup to register the genesis dump /// directly as finalized (bypassing pending). Fails if a finalized /// row already exists (the singleton constraint). - pub fn insert_finalized_dump( + /// Test seed only: production registers the genesis/recovery snapshot + /// through `insert_initial_finalized_dump`, which binds the canonical + /// coordinates atomically (this branch replaced both former + /// production callers). + #[cfg(test)] + pub(crate) fn insert_finalized_dump( + &mut self, + prefix: &Path, + inclusion_block: u64, + l2_tx_index: u64, + ) -> Result { + self.write(|tx| { + let executed_input_count = next_executed_input_count_in(tx)?; + insert_finalized_dump_in( + tx, + prefix, + inclusion_block, + l2_tx_index, + executed_input_count, + ) + }) + } + + /// Establish the era's application-history base, durable safe-input drain + /// floor, and initial finalized snapshot in one transaction. Plain setup + /// reasserts the migration's zero bases; cockroach setup binds the folded + /// application's absolute executed-input count and the recovery root's + /// exclusive safe-input cursor for the first and only time. + pub(crate) fn insert_initial_finalized_dump( &mut self, prefix: &Path, inclusion_block: u64, l2_tx_index: u64, + base_executed_input_count: u64, + base_safe_input_index: u64, ) -> Result { self.write(|tx| { - tx.execute( - "INSERT INTO dumps (prefix) VALUES (?1)", - params![path_to_text(prefix)], - )?; - let dump_id = tx.last_insert_rowid(); - tx.execute( - "INSERT INTO finalized_snapshot \ - (singleton_id, dump_id, inclusion_block, l2_tx_index) \ - VALUES (0, ?1, ?2, ?3)", - params![ - dump_id, - u64_to_i64(inclusion_block), - u64_to_i64(l2_tx_index) - ], - )?; - Ok(dump_id) + bind_history_base_in(tx, base_executed_input_count, base_safe_input_index)?; + insert_finalized_dump_in( + tx, + prefix, + inclusion_block, + l2_tx_index, + ExecutedInputCount::new(base_executed_input_count), + ) }) } @@ -387,6 +478,45 @@ impl Storage { } } +fn assert_snapshot_count_in( + tx: &Transaction<'_>, + executed_input_count: ExecutedInputCount, +) -> Result<()> { + assert_eq!( + executed_input_count, + next_executed_input_count_in(tx)?, + "snapshot executed-input count differs from canonical storage history" + ); + Ok(()) +} + +fn insert_finalized_dump_in( + tx: &Transaction<'_>, + prefix: &Path, + inclusion_block: u64, + l2_tx_index: u64, + executed_input_count: ExecutedInputCount, +) -> Result { + assert_snapshot_count_in(tx, executed_input_count)?; + tx.execute( + "INSERT INTO dumps (prefix) VALUES (?1)", + params![path_to_text(prefix)], + )?; + let dump_id = tx.last_insert_rowid(); + tx.execute( + "INSERT INTO finalized_snapshot \ + (singleton_id, dump_id, inclusion_block, l2_tx_index, executed_input_count) \ + VALUES (0, ?1, ?2, ?3, ?4)", + params![ + dump_id, + u64_to_i64(inclusion_block), + u64_to_i64(l2_tx_index), + u64_to_i64(executed_input_count.get()), + ], + )?; + Ok(dump_id) +} + // ── transaction-scoped helpers ──────────────────────────────────────────── pub(super) fn insert_pending_dump_in( @@ -394,16 +524,24 @@ pub(super) fn insert_pending_dump_in( prefix: &Path, nonce: u64, l2_tx_index: u64, + executed_input_count: ExecutedInputCount, ) -> Result { + assert_snapshot_count_in(tx, executed_input_count)?; tx.execute( "INSERT INTO dumps (prefix) VALUES (?1)", params![path_to_text(prefix)], )?; let dump_id = tx.last_insert_rowid(); tx.execute( - "INSERT INTO pending_snapshots (nonce, dump_id, l2_tx_index) \ - VALUES (?1, ?2, ?3)", - params![u64_to_i64(nonce), dump_id, u64_to_i64(l2_tx_index)], + "INSERT INTO pending_snapshots \ + (nonce, dump_id, l2_tx_index, executed_input_count) \ + VALUES (?1, ?2, ?3, ?4)", + params![ + u64_to_i64(nonce), + dump_id, + u64_to_i64(l2_tx_index), + u64_to_i64(executed_input_count.get()), + ], )?; Ok(dump_id) } @@ -415,17 +553,23 @@ pub(super) fn promote_finalized_in( ) -> Result<()> { // The promoted dump's bytes correspond to state at batch close, so // we carry over its `l2_tx_index` directly. - let (new_dump_id, l2_tx_index): (i64, i64) = tx.query_row( - "SELECT dump_id, l2_tx_index FROM pending_snapshots WHERE nonce = ?1", + let (new_dump_id, l2_tx_index, executed_input_count): (i64, i64, i64) = tx.query_row( + "SELECT dump_id, l2_tx_index, executed_input_count \ + FROM pending_snapshots WHERE nonce = ?1", params![u64_to_i64(max_nonce)], - |row| Ok((row.get(0)?, row.get(1)?)), + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), )?; tx.execute( "INSERT OR REPLACE INTO finalized_snapshot \ - (singleton_id, dump_id, inclusion_block, l2_tx_index) \ - VALUES (0, ?1, ?2, ?3)", - params![new_dump_id, u64_to_i64(inclusion_block), l2_tx_index], + (singleton_id, dump_id, inclusion_block, l2_tx_index, executed_input_count) \ + VALUES (0, ?1, ?2, ?3, ?4)", + params![ + new_dump_id, + u64_to_i64(inclusion_block), + l2_tx_index, + executed_input_count, + ], )?; // Clean up the promoted row plus any stale ones behind it. The @@ -488,17 +632,21 @@ fn delete_dump_row_in(tx: &Transaction<'_>, dump_id: i64) -> Result<()> { fn latest_pending_dump_in(tx: &Transaction<'_>) -> Result> { tx.query_row( - "SELECT p.nonce, p.dump_id, d.prefix, p.l2_tx_index \ + "SELECT p.nonce, p.dump_id, d.prefix, p.l2_tx_index, p.executed_input_count \ FROM pending_snapshots p \ - JOIN dumps d ON d.id = p.dump_id \ + LEFT JOIN dumps d ON d.id = p.dump_id \ ORDER BY p.nonce DESC \ LIMIT 1", [], |row| { let nonce: i64 = row.get(0)?; let dump_id: i64 = row.get(1)?; + // LEFT JOIN keeps a dangling reference visible; reading NULL as + // String then returns InvalidColumnType instead of laundering the + // corruption into OptionalExtension's `None`. let prefix: String = row.get(2)?; let l2_tx_index: i64 = row.get(3)?; + let executed_input_count: i64 = row.get(4)?; Ok(PendingDump { nonce: i64_to_u64(nonce), dump: DumpRow { @@ -506,6 +654,7 @@ fn latest_pending_dump_in(tx: &Transaction<'_>) -> Result> { prefix: PathBuf::from(prefix), }, l2_tx_index: i64_to_u64(l2_tx_index), + executed_input_count: ExecutedInputCount::new(i64_to_u64(executed_input_count)), }) }, ) @@ -514,16 +663,20 @@ fn latest_pending_dump_in(tx: &Transaction<'_>) -> Result> { fn finalized_dump_in(tx: &Transaction<'_>) -> Result> { tx.query_row( - "SELECT f.dump_id, d.prefix, f.inclusion_block, f.l2_tx_index \ + "SELECT f.dump_id, d.prefix, f.inclusion_block, f.l2_tx_index, \ + f.executed_input_count \ FROM finalized_snapshot f \ - JOIN dumps d ON d.id = f.dump_id \ + LEFT JOIN dumps d ON d.id = f.dump_id \ WHERE f.singleton_id = 0", [], |row| { let dump_id: i64 = row.get(0)?; + // See latest_pending_dump_in: a missing referenced dump row must + // be an error, not an apparent absence of the singleton. let prefix: String = row.get(1)?; let inclusion_block: i64 = row.get(2)?; let l2_tx_index: i64 = row.get(3)?; + let executed_input_count: i64 = row.get(4)?; Ok(FinalizedDump { dump: DumpRow { id: dump_id, @@ -531,6 +684,7 @@ fn finalized_dump_in(tx: &Transaction<'_>) -> Result> { }, inclusion_block: i64_to_u64(inclusion_block), l2_tx_index: i64_to_u64(l2_tx_index), + executed_input_count: ExecutedInputCount::new(i64_to_u64(executed_input_count)), }) }, ) @@ -542,10 +696,14 @@ fn finalized_dump_in(tx: &Transaction<'_>) -> Result> { /// resume checkpoint) and [`Storage::acquire_latest_snapshot_lease`] (the /// `/latest_snapshot` lease), so the "pending else finalized" rule lives in one /// place. -fn latest_snapshot_in(tx: &Transaction<'_>) -> Result> { +fn latest_snapshot_in(tx: &Transaction<'_>) -> Result> { Ok(match latest_pending_dump_in(tx)? { - Some(pending) => Some((pending.dump, pending.l2_tx_index)), - None => finalized_dump_in(tx)?.map(|f| (f.dump, f.l2_tx_index)), + Some(pending) => Some(( + pending.dump, + pending.l2_tx_index, + pending.executed_input_count, + )), + None => finalized_dump_in(tx)?.map(|f| (f.dump, f.l2_tx_index, f.executed_input_count)), }) } @@ -568,7 +726,7 @@ pub(super) fn clear_pending_dumps_in(tx: &Transaction<'_>) -> Result { /// Scoped pending-snapshot clear for the recovery cascade: delete only the /// rows whose `nonce >= from_nonce` (the cascade pivot's nonce) — exactly /// the cascaded batches' pendings. Lower-nonce rows are gold-but-unpromoted -/// pendings that must survive (review F9: deleting them arms a +/// pendings that must survive (deleting them arms a /// promote-wedge crash-loop when their landing is later observed). Same /// same-transaction composition rationale as [`clear_pending_dumps_in`]. pub(super) fn clear_pending_dumps_from_nonce_in( @@ -614,10 +772,12 @@ fn path_to_text(path: &Path) -> String { mod tests { use std::collections::HashSet; use std::path::PathBuf; + use std::sync::Arc; + use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; - use crate::storage::{Storage, test_helpers::temp_db}; + use crate::storage::{ExecutedInputCount, LifecycleCommand, Storage, test_helpers::temp_db}; - use super::{DumpRow, FinalizedDump, PendingDump}; + use super::{DumpRow, FinalizedDump, LeaseGuard, PendingDump}; fn prefix(n: u64) -> PathBuf { PathBuf::from(format!("/data/dumps/{n}")) @@ -642,6 +802,7 @@ mod tests { prefix: prefix(0), }, l2_tx_index: 10, + executed_input_count: ExecutedInputCount::ZERO, } ); @@ -667,6 +828,96 @@ mod tests { ); } + #[test] + fn initial_finalized_snapshot_binds_rebuild_base_atomically() { + let db = temp_db("initial-finalized-history-base"); + let mut storage = + Storage::initialize_for_command(db.path.as_str(), LifecycleCommand::Rebuild) + .expect("initialize rebuild"); + assert_eq!( + storage + .history_state() + .expect("pending history") + .base_executed_input_count, + None + ); + assert_eq!( + storage + .history_state() + .expect("pending history") + .base_safe_input_index, + None + ); + + storage + .conn + .execute_batch( + "CREATE TRIGGER fail_initial_finalized_snapshot + BEFORE INSERT ON finalized_snapshot + BEGIN + SELECT RAISE(ABORT, 'injected finalized snapshot failure'); + END;", + ) + .expect("install failure trigger"); + let err = storage + .insert_initial_finalized_dump(&prefix(0), 100, 7, 41, 7) + .expect_err("snapshot failure must roll back the history base"); + assert!( + err.to_string() + .contains("injected finalized snapshot failure"), + "unexpected error: {err:?}" + ); + assert_eq!( + storage + .history_state() + .expect("history after rollback") + .base_executed_input_count, + None, + "the base cannot survive without its establishing snapshot" + ); + assert_eq!( + storage + .history_state() + .expect("history after rollback") + .base_safe_input_index, + None, + "the safe-input floor cannot survive without its establishing snapshot" + ); + assert!(storage.finalized_dump().expect("read finalized").is_none()); + assert!(storage.list_dump_rows().expect("read dumps").is_empty()); + + storage + .conn + .execute_batch("DROP TRIGGER fail_initial_finalized_snapshot;") + .expect("remove failure trigger"); + storage + .insert_initial_finalized_dump(&prefix(1), 100, 7, 41, 7) + .expect("bind base with finalized snapshot"); + assert_eq!( + storage + .history_state() + .expect("bound history") + .base_executed_input_count, + Some(41) + ); + assert_eq!( + storage + .history_state() + .expect("bound history") + .base_safe_input_index, + Some(7) + ); + assert_eq!( + storage + .finalized_dump() + .expect("read finalized") + .expect("finalized snapshot") + .l2_tx_index, + 7, + "the physical replay cursor remains distinct from application base K" + ); + } + #[test] fn latest_pending_picks_highest_nonce() { let db = temp_db("latest-pending"); @@ -705,6 +956,7 @@ mod tests { }, inclusion_block: 500, l2_tx_index: 102, + executed_input_count: ExecutedInputCount::ZERO, } ); @@ -1032,11 +1284,74 @@ mod tests { release(); } + fn install_deferred_lease_commit_failure(path: &str) { + let conn = Storage::open_connection(path).expect("open failure-injection connection"); + conn.execute_batch( + "CREATE TABLE lease_commit_parent ( + id INTEGER PRIMARY KEY + ); + CREATE TABLE lease_commit_child ( + id INTEGER PRIMARY KEY, + parent_id INTEGER NOT NULL + REFERENCES lease_commit_parent(id) + DEFERRABLE INITIALLY DEFERRED + ); + CREATE TRIGGER fail_lease_commit + AFTER UPDATE OF lease_count ON dumps + WHEN NEW.lease_count > OLD.lease_count + BEGIN + INSERT INTO lease_commit_child(parent_id) VALUES (1); + END;", + ) + .expect("install deferred commit failure"); + } + + fn noop_reporter() -> super::PersistentReleaseFailureReporter { + Arc::new(|_cause: &str| {}) + } + + fn counting_scheduler(scheduled: Arc) -> super::ReleaseScheduler { + Arc::new(move |_release| { + scheduled.fetch_add(1, Ordering::SeqCst); + }) + } + + #[test] + fn lease_release_reports_persistent_missing_row_failure() { + let db = temp_db("lease-release-persistent-failure"); + let _storage = Storage::open(db.path.as_str()).expect("open"); + let reported = Arc::new(AtomicBool::new(false)); + let reporter = { + let reported = reported.clone(); + Arc::new(move |_cause: &str| { + reported.store(true, Ordering::SeqCst); + }) + }; + let guard = LeaseGuard { + path: db.path, + dump_id: i64::MAX, + schedule: Arc::new(inline), + report_persistent_failure: reporter, + }; + + drop(guard); + + assert!( + reported.load(Ordering::SeqCst), + "a durable lease-row invariant failure must reach the runtime reporter" + ); + } + #[test] fn acquire_finalized_lease_returns_none_when_no_finalized() { let db = temp_db("acquire-finalized-none"); let mut storage = Storage::open(db.path.as_str()).expect("open"); - assert!(storage.acquire_finalized_lease(inline).unwrap().is_none()); + assert!( + storage + .acquire_finalized_lease(Arc::new(inline), noop_reporter()) + .unwrap() + .is_none() + ); } #[test] @@ -1046,7 +1361,7 @@ mod tests { let id_a = storage.insert_finalized_dump(&prefix(0), 100, 5).unwrap(); let leased = storage - .acquire_finalized_lease(inline) + .acquire_finalized_lease(Arc::new(inline), noop_reporter()) .unwrap() .expect("a finalized snapshot exists"); assert_eq!(leased.prefix, prefix(0)); @@ -1090,6 +1405,37 @@ mod tests { assert!(eligible.contains(&id_a), "released dump is GC-eligible"); } + #[test] + fn failed_finalized_lease_commit_never_arms_a_release() { + let db = temp_db("acquire-finalized-commit-failure"); + let mut storage = Storage::open(db.path.as_str()).expect("open"); + let dump_id = storage.insert_finalized_dump(&prefix(0), 100, 5).unwrap(); + install_deferred_lease_commit_failure(db.path.as_str()); + let scheduled = Arc::new(AtomicUsize::new(0)); + + let err = match storage + .acquire_finalized_lease(counting_scheduler(scheduled.clone()), noop_reporter()) + { + Ok(_) => panic!("deferred foreign-key violation must fail COMMIT"), + Err(err) => err, + }; + + assert!( + err.to_string().contains("FOREIGN KEY"), + "expected deferred constraint failure, got: {err}" + ); + assert_eq!( + scheduled.load(Ordering::SeqCst), + 0, + "a rolled-back increment has no matching release to schedule" + ); + assert_eq!( + storage.dump_lease_count(dump_id).unwrap(), + Some(0), + "the failed transaction rolled back the lease increment" + ); + } + #[test] fn acquire_latest_snapshot_lease_prefers_pending_and_leases() { let db = temp_db("acquire-latest-pending"); @@ -1099,7 +1445,7 @@ mod tests { let id_pending = storage.insert_pending_dump(&prefix(1), 3, 9).unwrap(); let leased = storage - .acquire_latest_snapshot_lease(inline) + .acquire_latest_snapshot_lease(Arc::new(inline), noop_reporter()) .unwrap() .expect("a snapshot exists"); assert_eq!(leased.prefix, prefix(1), "prefers the latest pending"); @@ -1137,10 +1483,41 @@ mod tests { storage.insert_finalized_dump(&prefix(0), 100, 5).unwrap(); let leased = storage - .acquire_latest_snapshot_lease(inline) + .acquire_latest_snapshot_lease(Arc::new(inline), noop_reporter()) .unwrap() .expect("falls back to finalized"); assert_eq!(leased.prefix, prefix(0)); assert_eq!(leased.l2_tx_index, 5); } + + #[test] + fn failed_latest_snapshot_lease_commit_never_arms_a_release() { + let db = temp_db("acquire-latest-commit-failure"); + let mut storage = Storage::open(db.path.as_str()).expect("open"); + let dump_id = storage.insert_pending_dump(&prefix(0), 3, 9).unwrap(); + install_deferred_lease_commit_failure(db.path.as_str()); + let scheduled = Arc::new(AtomicUsize::new(0)); + + let err = match storage + .acquire_latest_snapshot_lease(counting_scheduler(scheduled.clone()), noop_reporter()) + { + Ok(_) => panic!("deferred foreign-key violation must fail COMMIT"), + Err(err) => err, + }; + + assert!( + err.to_string().contains("FOREIGN KEY"), + "expected deferred constraint failure, got: {err}" + ); + assert_eq!( + scheduled.load(Ordering::SeqCst), + 0, + "a rolled-back increment has no matching release to schedule" + ); + assert_eq!( + storage.dump_lease_count(dump_id).unwrap(), + Some(0), + "the failed transaction rolled back the lease increment" + ); + } } diff --git a/sequencer/src/storage/test_helpers.rs b/sequencer/src/storage/test_helpers.rs index 71393b9b..69a5830e 100644 --- a/sequencer/src/storage/test_helpers.rs +++ b/sequencer/src/storage/test_helpers.rs @@ -8,7 +8,7 @@ use sequencer_core::l2_tx::SequencedL2Tx; use sequencer_core::protocol::ProtocolTiming; use tempfile::TempDir; -use super::{SafeInputRange, Storage, StoredSafeInput}; +use super::{DeploymentIdentity, FeeOracleIdentity, SafeInputRange, Storage, StoredSafeInput}; pub(crate) const SENDER_A: Address = Address::repeat_byte(0xAA); pub(crate) const SENDER_B: Address = Address::repeat_byte(0xBB); @@ -42,9 +42,37 @@ pub(crate) fn temp_db(name: &str) -> TestDb { } } +/// Pin the trusted deployment identity a production setup establishes before +/// any fresh-Tip attribution can classify safe inputs. +pub(crate) fn pin_test_deployment_identity( + storage: &mut Storage, + batch_submitter_address: Address, +) { + storage + .load_or_insert_deployment_identity(DeploymentIdentity { + chain_id: 1, + app_address: Address::repeat_byte(0x11), + input_box_address: Address::repeat_byte(0x22), + app_deployment_block: 0, + batch_submitter_address, + fee_oracle: FeeOracleIdentity::Fixed { log_gas_price: 0 }, + }) + .expect("pin test deployment identity"); +} + +/// Recovery-storage fixture with the same prerequisite ordering as setup: +/// baseline schema first, then the persisted deployment identity, then any L1 +/// facts or batch-tree mutations driven by the individual test. +pub(crate) fn temp_db_with_default_deployment_identity(name: &str) -> TestDb { + let db = temp_db(name); + let mut storage = Storage::open(db.path.as_str()).expect("open test storage"); + pin_test_deployment_identity(&mut storage, SENDER_A); + db +} + /// Wire bytes of the local valid closed batch at `nonce` — the /// production-faithful "our batch landed on L1" payload. Hash-matches the -/// seal-time stamp, so the content-identity check (review R2) accepts it. +/// seal-time stamp, so the content-identity check accepts it. /// Panics if no valid closed local batch carries `nonce` (test bug: an /// accepted landing without a matching local batch is, by design, a /// canonical divergence). @@ -128,7 +156,7 @@ pub(crate) fn all_ordered_l2_txs(storage: &mut Storage) -> Vec { .ordered_l2_txs_page_from(0, 1_000_000) .expect("load all ordered l2 txs") .into_iter() - .map(|(_offset, tx, _frame_safe_block)| tx) + .map(|row| row.tx) .collect() } @@ -143,3 +171,25 @@ pub(crate) fn make_stale_batch_payload(nonce: u64, safe_block: u64) -> Vec { }], }) } + +/// Plant the canonical-divergence marker. Test-only lever for +/// I15 scenarios; production writes it only inside the content-identity +/// check's sync transaction. +pub(crate) fn record_canonical_divergence( + storage: &mut Storage, + nonce: u64, + safe_input_index: u64, +) { + storage + .conn + .execute( + "INSERT INTO canonical_divergence \ + (singleton_id, nonce, safe_input_index, kind, detected_at_ms) \ + VALUES (0, ?1, ?2, 'mismatch', 0)", + [ + i64::try_from(nonce).unwrap(), + i64::try_from(safe_input_index).unwrap(), + ], + ) + .expect("record canonical divergence"); +} diff --git a/tests/benchmarks/README.md b/tests/benchmarks/README.md index caa2ae4e..bfdebc39 100644 --- a/tests/benchmarks/README.md +++ b/tests/benchmarks/README.md @@ -74,6 +74,7 @@ cargo run -p benchmarks --bin compare_latest --release -- --results-dir tests/be ## Notes - Self-contained variants launch `anvil --load-state` from the preloaded rollups dump under `tests/.deps/`; run `just setup` first. +- Self-contained sweeps run the load clients and sequencer on the same machine. At high concurrency the clients can starve the sequencer, producing a throughput plateau and rising latency without offering more usable server load. Treat this as host-level regression evidence, not the sequencer's capacity ceiling; use a separate-machine load generator for capacity measurement. - Self-contained variants also deploy a local `Application` through `ApplicationFactory`, so they require a canonical machine image at `examples/canonical-app/out/canonical-machine-image`; run `just canonical-build-machine-image` first. - Self-contained variants therefore require Foundry's `anvil` binary to be installed locally. - `--max-fee` must be at or above the placeholder app's base fee, or every tx is rejected (`422 EXECUTION_REJECTED`) and the run reports no accepted txs. The error message includes the rejection breakdown and the first rejection body, which names the base fee. diff --git a/tests/e2e/src/test_cases.rs b/tests/e2e/src/test_cases.rs index 35fd5cf2..a9f583de 100644 --- a/tests/e2e/src/test_cases.rs +++ b/tests/e2e/src/test_cases.rs @@ -45,6 +45,14 @@ const ACCEPTANCE_POLL_ATTEMPTS: usize = 40; /// Per-attempt pause while waiting for batch submission and safe-head ingestion. const ACCEPTANCE_POLL_INTERVAL: Duration = Duration::from_secs(1); +/// Product-visible live frame-clock interval. E2E advancement must derive from +/// the protocol constant rather than incidental Anvil transaction counts. +const FRAME_CLOCK_INTERVAL_SAFE_BLOCKS: u64 = + sequencer_core::protocol::ProtocolTiming::FRAME_CLOCK_INTERVAL_SAFE_BLOCKS; + +const FRAME_CLOCK_POLL_ATTEMPTS: usize = 80; +const FRAME_CLOCK_POLL_INTERVAL: Duration = Duration::from_millis(250); + // ── Zone-math constants for the outage-matrix and recovery tests ───────── // // These derive from the sequencer's default config so a change to @@ -174,8 +182,8 @@ pub fn test_cases() -> Vec<(&'static str, ScenarioFn)> { ("concurrent_user_ops_test", |runtime| { Box::pin(run_concurrent_user_ops_test(runtime)) }), - ("multi_deposit_same_block_test", |runtime| { - Box::pin(run_multi_deposit_same_block_test(runtime)) + ("multi_deposit_reconciliation_test", |runtime| { + Box::pin(run_multi_deposit_reconciliation_test(runtime)) }), ( "restart_after_committed_tx_replays_cleanly_test", @@ -199,9 +207,20 @@ pub fn test_cases() -> Vec<(&'static str, ScenarioFn)> { ("provider_outage_wall_clock_refuses_boot_test", |runtime| { Box::pin(run_provider_outage_wall_clock_refuses_boot_test(runtime)) }), - ("wall_clock_backward_jump_no_panic_test", |runtime| { - Box::pin(run_wall_clock_backward_jump_no_panic_test(runtime)) - }), + ( + "warm_restart_from_fresh_persisted_facts_with_l1_down_test", + |runtime| { + Box::pin(run_warm_restart_from_fresh_persisted_facts_with_l1_down_test(runtime)) + }, + ), + ( + "wall_clock_backward_jump_retries_then_recovers_test", + |runtime| { + Box::pin(run_wall_clock_backward_jump_retries_then_recovers_test( + runtime, + )) + }, + ), ("stalled_safe_head_startup_refuses_boot_test", |runtime| { Box::pin(run_stalled_safe_head_startup_refuses_boot_test(runtime)) }), @@ -380,7 +399,8 @@ async fn prepare_non_genesis_watchdog_state(runtime: &mut ManagedSequencer) -> S let withdrawal_amount = U256::from(150_000_u64); let gas = fee_to_linear(DEFAULT_FRAME_FEE); - apply_safe_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount).await?; + apply_reconciled_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount) + .await?; alice_l2.transfer(bob_address, transfer_amount).await?; replay.apply(ws.expect_user_op_from(alice_address).await?)?; @@ -417,16 +437,16 @@ async fn prepare_non_genesis_watchdog_state(runtime: &mut ManagedSequencer) -> S /// Mine L1 forward until a finalized snapshot is promoted at an inclusion block /// strictly above `floor` (the value observed before the batch that should /// promote), polling the DB instead of sleeping a fixed submitter-tick -/// interval. Each attempt mines a couple of blocks and pauses briefly, so the -/// submitter and promoter get to run; returns the new inclusion block, or times -/// out loudly. Robust against CI jitter — it waits exactly as long as promotion -/// takes, no more. +/// interval. Each attempt mines one live block and pauses for the same wall +/// time, so the submitter and promoter get to run without synthesizing a future +/// L1 clock; returns the new inclusion block, or times out loudly. Robust +/// against CI jitter — it waits exactly as long as promotion takes, no more. async fn mine_until_finalized_advances( runtime: &ManagedSequencer, floor: u64, ) -> ScenarioResult { for _ in 0..PROMOTION_POLL_ATTEMPTS { - runtime.mine_l1_blocks(2).await?; + runtime.mine_live_l1_blocks(1).await?; tokio::time::sleep(PROMOTION_POLL_INTERVAL).await; let (inclusion_block, _) = runtime.finalized_snapshot_info()?; if inclusion_block > floor { @@ -443,7 +463,7 @@ async fn mine_until_batch_is_safe_accepted( runtime: &ManagedSequencer, ) -> ScenarioResult<(u64, Option)> { for _ in 0..ACCEPTANCE_POLL_ATTEMPTS { - runtime.mine_l1_blocks(2).await?; + runtime.mine_live_l1_blocks(1).await?; tokio::time::sleep(ACCEPTANCE_POLL_INTERVAL).await; let accepted = runtime.count_safe_accepted_batches()?; if accepted.0 > 0 { @@ -453,6 +473,87 @@ async fn mine_until_batch_is_safe_accepted( Err("timed out waiting for a batch to reach safe_accepted_batches".into()) } +/// Advance Anvil only as far as the next semantic live-frame tick needs, then +/// wait for the input reader and inclusion lane to commit a frame covering +/// `required_safe_block`. The live safe-head query prevents stale SQLite +/// observations from turning an asynchronous reader poll into over-mining. +async fn advance_live_frame_until_covers( + runtime: &ManagedSequencer, + required_safe_block: u64, +) -> ScenarioResult { + for _ in 0..FRAME_CLOCK_POLL_ATTEMPTS { + let (frame_safe_block, _) = runtime.frame_clock_observation()?; + if frame_safe_block >= required_safe_block { + return Ok(frame_safe_block); + } + + let next_tick = frame_safe_block + .checked_add(FRAME_CLOCK_INTERVAL_SAFE_BLOCKS) + .ok_or("frame-clock target overflow")?; + let target_safe_head = next_tick.max(required_safe_block); + let live_safe_head = runtime.l1_safe_block_number().await?; + if live_safe_head < target_safe_head { + runtime + .mine_live_l1_blocks(target_safe_head - live_safe_head) + .await?; + } + + tokio::time::sleep(FRAME_CLOCK_POLL_INTERVAL).await; + } + + let (frame_safe_block, persisted_safe_head) = runtime.frame_clock_observation()?; + let live_safe_head = runtime.l1_safe_block_number().await?; + Err(format!( + "timed out advancing live frame to cover block {required_safe_block}: \ + frame={frame_safe_block} persisted_safe_head={persisted_safe_head} \ + live_safe_head={live_safe_head}" + ) + .into()) +} + +/// Reach one exact Anvil safe head and wait for the input reader to persist it. +/// Used only by the frame-clock boundary test, where overshooting would erase +/// the below-threshold assertion. +async fn advance_persisted_safe_head_exactly( + runtime: &ManagedSequencer, + target_safe_head: u64, +) -> ScenarioResult<()> { + for _ in 0..FRAME_CLOCK_POLL_ATTEMPTS { + let live_safe_head = runtime.l1_safe_block_number().await?; + if live_safe_head > target_safe_head { + return Err(format!( + "cannot pin safe head {target_safe_head}: live Anvil safe head is already {live_safe_head}" + ) + .into()); + } + if live_safe_head < target_safe_head { + // Mine one at a time: Anvil's safe tag may lag the chain head, so + // requested block count and safe-head delta are not interchangeable. + runtime.mine_live_l1_blocks(1).await?; + tokio::time::sleep(FRAME_CLOCK_POLL_INTERVAL).await; + continue; + } + + let (_, persisted_safe_head) = runtime.frame_clock_observation()?; + if persisted_safe_head == target_safe_head { + return Ok(()); + } + if persisted_safe_head > target_safe_head { + return Err(format!( + "persisted safe head overshot {target_safe_head}: got {persisted_safe_head}" + ) + .into()); + } + tokio::time::sleep(FRAME_CLOCK_POLL_INTERVAL).await; + } + + let (_, persisted_safe_head) = runtime.frame_clock_observation()?; + Err(format!( + "timed out waiting for persisted safe head {target_safe_head}: got {persisted_safe_head}" + ) + .into()) +} + /// Close the open batch, land it on L1, and wait for snapshot promotion so /// `/finalized_state` reports a new `inclusion_block`. async fn drive_finalized_gold_batch_for_watchdog( @@ -496,7 +597,7 @@ async fn run_direct_input_not_safe_yet_test(runtime: &mut ManagedSequencer) -> S let transfer_amount = U256::from(400_000_u64); let gas = fee_to_linear(DEFAULT_FRAME_FEE); - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay, @@ -505,21 +606,68 @@ async fn run_direct_input_not_safe_yet_test(runtime: &mut ManagedSequencer) -> S ) .await?; + let (frame_anchor, persisted_safe_head) = runtime.frame_clock_observation()?; + let live_safe_head = runtime.l1_safe_block_number().await?; + assert_eq!( + persisted_safe_head, frame_anchor, + "funding reconciliation must leave a clean frame-clock anchor", + ); + assert_eq!( + live_safe_head, frame_anchor, + "Anvil safe head must match the clean frame-clock anchor", + ); + alice_l1 .mint_supported_token(pending_deposit_amount) .await?; - alice_l1 + let deposit_block = alice_l1 .deposit_supported_token(pending_deposit_amount) .await?; alice_l2.transfer(bob_address, transfer_amount).await?; - replay.apply(ws.expect_user_op_from(alice_address).await?)?; + let user_op = ws.expect_user_op_from(alice_address).await?; + match &user_op { + WsTxMessage::UserOp { safe_block, .. } => assert_eq!( + *safe_block, frame_anchor, + "below-threshold direct must not advance the user-op frame clock", + ), + other => unreachable!("expected user op after typed WS assertion, got {other:?}"), + } + replay.apply(user_op)?; - runtime.mine_l1_blocks(1).await?; - replay.apply( - ws.expect_direct_input_from(runtime.erc20_portal_address()) - .await?, - )?; + let below_threshold = frame_anchor + FRAME_CLOCK_INTERVAL_SAFE_BLOCKS - 1; + assert!( + deposit_block <= below_threshold, + "boundary setup consumed too many L1 blocks: deposit={deposit_block} \ + below_threshold={below_threshold}", + ); + advance_persisted_safe_head_exactly(runtime, below_threshold).await?; + assert_eq!( + runtime.frame_clock_observation()?, + (frame_anchor, below_threshold), + "S+4 must leave the open frame at S", + ); + ws.expect_no_message_for(NO_WS_MESSAGE_WAIT).await?; + + let at_threshold = frame_anchor + FRAME_CLOCK_INTERVAL_SAFE_BLOCKS; + advance_persisted_safe_head_exactly(runtime, at_threshold).await?; + let rotated_frame = advance_live_frame_until_covers(runtime, deposit_block).await?; + assert_eq!( + rotated_frame, at_threshold, + "S+5 must create exactly one frame at the observed safe head", + ); + let direct = ws + .expect_direct_input_from(runtime.erc20_portal_address()) + .await?; + match &direct { + WsTxMessage::DirectInput { block_number, .. } => assert_eq!( + *block_number, deposit_block, + "direct must retain its exact L1 inclusion block", + ), + other => unreachable!("expected direct after typed WS assertion, got {other:?}"), + } + replay.apply(direct)?; + ws.expect_no_message_for(NO_WS_MESSAGE_WAIT).await?; // Alice: 700_000 - 400_000 - gas + 600_000. Bob: 400_000 (no ops from Bob). assert_wallet_state( @@ -557,7 +705,8 @@ async fn run_rejected_user_op_not_broadcast_test( let transfer_amount = U256::from(100_000_u64); let gas = fee_to_linear(DEFAULT_FRAME_FEE); - apply_safe_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount).await?; + apply_reconciled_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount) + .await?; alice_l2.transfer(bob_address, transfer_amount).await?; replay.apply(ws.expect_user_op_from(alice_address).await?)?; @@ -608,9 +757,14 @@ async fn run_reconnect_from_offset_test(runtime: &mut ManagedSequencer) -> Scena let withdrawal_amount = U256::from(100_000_u64); let gas = fee_to_linear(DEFAULT_FRAME_FEE); - let deposit_message = - apply_safe_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount) - .await?; + let deposit_message = apply_reconciled_supported_deposit( + runtime, + &mut ws, + &mut replay, + &alice_l1, + deposit_amount, + ) + .await?; // WS replay is cursor-based and exclusive: `from_offset` means // "start after this already-consumed DB offset". let reconnect_offset = deposit_message.offset(); @@ -658,7 +812,7 @@ async fn run_restart_and_replay_test(runtime: &mut ManagedSequencer) -> Scenario let withdrawal_amount = U256::from(150_000_u64); let gas = fee_to_linear(DEFAULT_FRAME_FEE); - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before_restart, @@ -740,15 +894,22 @@ async fn run_unsupported_token_deposit_noop_test( alice_l1 .mint_token(unsupported_token, U256::from(123_000_u64)) .await?; - alice_l1 + let deposit_block = alice_l1 .deposit_token(unsupported_token, U256::from(123_000_u64)) .await?; - runtime.mine_l1_blocks(1).await?; + advance_live_frame_until_covers(runtime, deposit_block).await?; - replay.apply( - ws.expect_direct_input_from(runtime.erc20_portal_address()) - .await?, - )?; + let direct = ws + .expect_direct_input_from(runtime.erc20_portal_address()) + .await?; + match &direct { + WsTxMessage::DirectInput { block_number, .. } => assert_eq!( + *block_number, deposit_block, + "unsupported-token direct must retain its L1 inclusion block", + ), + other => unreachable!("expected direct after typed WS assertion, got {other:?}"), + } + replay.apply(direct)?; assert_eq!(replay.current_user_balance(alice_address), U256::ZERO); assert_eq!(replay.current_user_nonce(alice_address), 0); @@ -756,7 +917,7 @@ async fn run_unsupported_token_deposit_noop_test( Ok(()) } -async fn apply_safe_supported_deposit( +async fn apply_reconciled_supported_deposit( runtime: &ManagedSequencer, ws: &mut WsClient, replay: &mut ReplayWalletApp, @@ -764,12 +925,19 @@ async fn apply_safe_supported_deposit( amount: U256, ) -> ScenarioResult { wallet_l1.mint_supported_token(amount).await?; - wallet_l1.deposit_supported_token(amount).await?; - runtime.mine_l1_blocks(1).await?; + let deposit_block = wallet_l1.deposit_supported_token(amount).await?; + advance_live_frame_until_covers(runtime, deposit_block).await?; let message = ws .expect_direct_input_from(runtime.erc20_portal_address()) .await?; + match &message { + WsTxMessage::DirectInput { block_number, .. } => assert_eq!( + *block_number, deposit_block, + "reconciled direct must retain its exact L1 inclusion block", + ), + other => unreachable!("expected direct after typed WS assertion, got {other:?}"), + } replay.apply(message.clone())?; Ok(message) } @@ -783,7 +951,8 @@ async fn run_fee_below_minimum_rejected_test(runtime: &mut ManagedSequencer) -> let mut replay = ReplayWalletApp::devnet(); let deposit_amount = U256::from(600_000_u64); - apply_safe_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount).await?; + apply_reconciled_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount) + .await?; // Submit user-op with max_fee=0, which is below the default frame fee (1356). let client = SequencerClient::new(runtime.endpoint())?; @@ -834,7 +1003,8 @@ async fn run_fixed_fee_oracle_sets_frame_fee_test( let mut replay = ReplayWalletApp::devnet(); let deposit_amount = U256::from(600_000_u64); - apply_safe_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount).await?; + apply_reconciled_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount) + .await?; let transfer_amount = U256::from(100_u64); alice_l2.transfer(alice_address, transfer_amount).await?; @@ -897,7 +1067,8 @@ async fn run_concurrent_user_ops_test(runtime: &mut ManagedSequencer) -> Scenari // Fund all signers via L1 deposits. for signer in &signers { let l1 = runtime.wallet_l1(signer.clone()).await?; - apply_safe_supported_deposit(runtime, &mut ws, &mut replay, &l1, deposit_amount).await?; + apply_reconciled_supported_deposit(runtime, &mut ws, &mut replay, &l1, deposit_amount) + .await?; } // Submit transfers concurrently from all signers (each sends to signer 0). @@ -962,7 +1133,9 @@ async fn run_concurrent_user_ops_test(runtime: &mut ManagedSequencer) -> Scenari Ok(()) } -async fn run_multi_deposit_same_block_test(runtime: &mut ManagedSequencer) -> ScenarioResult<()> { +async fn run_multi_deposit_reconciliation_test( + runtime: &mut ManagedSequencer, +) -> ScenarioResult<()> { let alice = TestSigner::from_default(1)?; let bob = TestSigner::from_default(2)?; let alice_address = alice.address(); @@ -976,14 +1149,16 @@ async fn run_multi_deposit_same_block_test(runtime: &mut ManagedSequencer) -> Sc let alice_deposit = U256::from(500_000_u64); let bob_deposit = U256::from(300_000_u64); - // Mint and deposit for both in quick succession (before mining). - alice_l1 + // Default automining puts these deposits in separate blocks. This scenario + // pins one reconciliation turn draining multiple accumulated directs; a + // true same-block ordering scenario is tracked separately. + let alice_deposit_block = alice_l1 .mint_and_deposit_supported_token(alice_deposit) .await?; - bob_l1.mint_and_deposit_supported_token(bob_deposit).await?; + let bob_deposit_block = bob_l1.mint_and_deposit_supported_token(bob_deposit).await?; - // Mine to make both deposits safe. - runtime.mine_l1_blocks(1).await?; + // Drive one or more semantic frame ticks until both deposits are covered. + advance_live_frame_until_covers(runtime, alice_deposit_block.max(bob_deposit_block)).await?; // Expect two direct inputs (one per deposit). let portal = runtime.erc20_portal_address(); @@ -1017,7 +1192,8 @@ async fn run_restart_after_committed_tx_replays_cleanly_test( let transfer_amount = U256::from(100_000_u64); let gas = fee_to_linear(DEFAULT_FRAME_FEE); - apply_safe_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount).await?; + apply_reconciled_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount) + .await?; // Submit a transfer, then immediately restart. alice_l2.transfer(alice_address, transfer_amount).await?; @@ -1070,7 +1246,7 @@ async fn run_recovery_after_stale_batches_test( let gas = fee_to_linear(DEFAULT_FRAME_FEE); // Step 1: Fund Alice via L1 deposit. - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, @@ -1209,7 +1385,7 @@ async fn run_setup_recovery_round_trip_test(runtime: &mut ManagedSequencer) -> S // the post-recovery gold-batch pump (~150 self-transfers × ~38k fee). let deposit = U256::from(10_000_000_u64); let transfer1 = U256::from(100_000_u64); - apply_safe_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit).await?; + apply_reconciled_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit).await?; alice_l2.transfer(bob_address, transfer1).await?; replay.apply(ws.expect_user_op_from(alice_address).await?)?; let expected_alice = deposit - transfer1 - gas; @@ -1334,7 +1510,7 @@ async fn run_sequencer_outage_pre_danger_no_recovery_test( let gas = fee_to_linear(DEFAULT_FRAME_FEE); // Step 1: Fund Alice and record a transfer. - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, @@ -1404,7 +1580,7 @@ async fn run_sequencer_outage_pre_danger_no_recovery_test( // - `check_danger` returns `TipInDanger(idx)` — the closed-frontier check finds // nothing past gold, but the open Tip's first frame has aged past // `danger_threshold`. -// - `decide_startup_action` returns `RecoverTip` (no flush — the Tip has +// - the startup reducer selects `RecoverTip` (no flush — the Tip has // no L1 footprint). // - `recover_aging_tip` cascades the Tip; pre-outage soft-confirmed user // ops are rolled back (this is the documented "soft confirmations may @@ -1420,10 +1596,10 @@ async fn run_sequencer_outage_pre_danger_no_recovery_test( async fn run_sequencer_outage_danger_zone_tip_cascade_test( runtime: &mut ManagedSequencer, ) -> ScenarioResult<()> { - // Pick advance in the danger zone: > danger_threshold (900) but < MAX_WAIT (1200). - // Decoupled from wall clock on purpose: this test exercises the - // block-based danger check in isolation. Uses module-level - // `DANGER_ZONE_BLOCKS` (see top-of-file zone constants). + // Pick an advance in the danger zone: > danger_threshold (900) but < + // MAX_WAIT (1200). L1 block time and process time advance together so the + // observed Tip age selects `RecoverTip` without leaving the repaired L1 + // view future-dated during the mandatory post-recovery inspection. let alice = TestSigner::from_default(1)?; let bob = TestSigner::from_default(2)?; @@ -1439,7 +1615,7 @@ async fn run_sequencer_outage_danger_zone_tip_cascade_test( let transfer_amount = U256::from(100_000_u64); let gas = fee_to_linear(DEFAULT_FRAME_FEE); - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, @@ -1461,12 +1637,20 @@ async fn run_sequencer_outage_danger_zone_tip_cascade_test( // so the startup `RecoverTip` path cascades it (no flush — the Tip // has no L1 footprint). Alice's pre-outage transfer was a soft // confirmation against the Tip; it's rolled back. - runtime.mine_l1_blocks(DANGER_ZONE_BLOCKS).await?; + runtime + .advance_wall_and_mine(blocks_as_duration(DANGER_ZONE_BLOCKS)) + .await?; let _ = expected_alice_balance; let _ = expected_bob_balance; runtime.respawn().await?; + let counts = runtime.count_batches()?; + assert!( + counts.invalidated >= 1, + "RecoverTip must invalidate the aged Tip before admission: {counts:?}", + ); + // After Tip cascade: balances roll back to the post-deposit / pre-transfer // state, nonces reset, and the WS feed should not replay the invalidated // user op. @@ -1545,7 +1729,7 @@ async fn run_provider_outage_past_stale_cascades_test( let deposit_amount = U256::from(600_000_u64); let transfer_amount = U256::from(100_000_u64); - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, @@ -1575,9 +1759,9 @@ async fn run_provider_outage_past_stale_cascades_test( // Step 4: Respawn. The sequencer dials the proxy, the proxy forwards // to Anvil, `sync_to_current_safe_head` returns 1250+ blocks past the - // open Tip's first frame. `check_danger` fires `TipInDanger(idx)`, - // `decide_startup_action` returns `RecoverTip`, `recover_aging_tip` - // cascades the Tip and opens a fresh one. + // open Tip's first frame. `check_danger` fires `TipInDanger(idx)`, and + // the reducer's guarded `RecoverTip` phase cascades the Tip and opens a + // fresh one. runtime.respawn().await?; // Step 5: Verify via WS replay. @@ -1645,7 +1829,7 @@ async fn run_provider_outage_wall_clock_refuses_boot_test( let mut alice_l2 = runtime.wallet_l2(alice.clone())?; let mut replay_before = ReplayWalletApp::devnet(); - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, @@ -1673,7 +1857,8 @@ async fn run_provider_outage_wall_clock_refuses_boot_test( // - dials the proxy → sync_to_current_safe_head fails (L1 unreachable). // - sees the persisted safe block timestamp is older than the L1 // read-staleness threshold. - // - decide_startup_action returns Refuse(L1ViewStale) → process exits with failure. + // - the startup reducer returns Retry(L1ViewStale), so the process exits + // without runtime admission. let respawn_result = runtime.respawn().await; assert!( respawn_result.is_err(), @@ -1707,19 +1892,109 @@ async fn run_provider_outage_wall_clock_refuses_boot_test( Ok(()) } -// `SystemTime::now()` backward jump → `saturating_sub` handles -// cleanly, no panic. -// -// Scenario: normal setup creates DB state at real time T. Stop, disconnect -// proxy, backward-jump the clock via faketime, respawn with L1 unreachable. -// The wall-clock fallback runs: -// -// elapsed = now(T-1h).saturating_sub(last_sync_at_ms(≈T)) = 0 -// -// No danger → boot proceeds. After reconnect, normal operation resumes. -// If `saturating_sub` ever regresses to a plain subtraction (underflow -// panic on u64), this test panics at respawn. -async fn run_wall_clock_backward_jump_no_panic_test( +// Warm restart from a fresh persisted L1 view while the RPC is unreachable. +// +// This is the production-shaped discriminator for startup admission: an +// initial refresh attempt may fail, but a recent persisted view that still +// reduces to `Admit` is sufficient authority to prepare and launch the +// runtime. The test keeps the proxy disconnected through readiness, one full +// background-worker retry cadence, POST /tx, and WS broadcast. A boot path +// that makes provider reachability an unconditional admission gate fails at +// `respawn`; a runtime that launches without a usable inclusion lane fails the +// transaction or WS assertion. +async fn run_warm_restart_from_fresh_persisted_facts_with_l1_down_test( + runtime: &mut ManagedSequencer, +) -> ScenarioResult<()> { + const STABILITY_WINDOW: Duration = Duration::from_secs(3); + + let alice = TestSigner::from_default(1)?; + let bob = TestSigner::from_default(2)?; + let alice_address = alice.address(); + let bob_address = bob.address(); + let deposit_amount = U256::from(600_000_u64); + let transfer_amount = U256::from(100_000_u64); + let fee = fee_to_linear(DEFAULT_FRAME_FEE); + + // Persist a recent safe-head observation plus replayable wallet state, + // then stop cleanly so the next run boots over fresh facts without a + // recovery phase. + let alice_l1 = runtime.wallet_l1(alice.clone()).await?; + let mut ws = runtime.ws(0).await?; + let mut replay = ReplayWalletApp::devnet(); + apply_reconciled_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount) + .await?; + drop(ws); + runtime.stop().await?; + + // Route the next process through an already-disconnected gateway. The + // proxy stays down until after the admitted runtime serves the write and + // its corresponding ordered-tx feed event. + let proxy = TcpProxy::spawn(runtime.l1_endpoint()).await?; + runtime.set_l1_endpoint_override(Some(proxy.endpoint())); + proxy.disconnect(); + runtime.respawn().await?; + assert!( + !proxy.is_connected(), + "warm restart must reach readiness without reconnecting the L1 proxy", + ); + + // Cross the 2s input-reader/danger-detector cadence before submitting, so + // this proves more than racing a request ahead of the first failed RPC + // retry. Fresh local facts must keep the process admitted and live. + let exit = runtime.observe_for(STABILITY_WINDOW).await?; + assert!( + exit.is_none(), + "warm runtime must remain live while the fresh persisted L1 view is valid; got {exit:?}", + ); + + let mut ws_after = runtime.ws(0).await?; + let mut replay_after = ReplayWalletApp::devnet(); + replay_after.apply( + ws_after + .expect_direct_input_from(runtime.erc20_portal_address()) + .await?, + )?; + let mut alice_l2 = runtime.wallet_l2(alice)?; + alice_l2.transfer(bob_address, transfer_amount).await?; + replay_after.apply(ws_after.expect_user_op_from(alice_address).await?)?; + + assert_wallet_state( + &replay_after, + ExpectedWalletState { + address: alice_address, + balance: deposit_amount - transfer_amount - fee, + nonce: 1, + }, + ExpectedWalletState { + address: bob_address, + balance: transfer_amount, + nonce: 0, + }, + 2, + ); + assert!( + !proxy.is_connected(), + "POST /tx and WS must succeed before L1 connectivity is restored", + ); + assert_eq!( + runtime.count_batches()?.invalidated, + 0, + "a fresh-view warm restart must not invalidate the batch tree", + ); + + proxy.reconnect(); + runtime.set_l1_endpoint_override(None); + proxy.shutdown().await?; + Ok(()) +} + +// A wall-clock regression of at least one configured block interval makes a +// persisted L1-progress timestamp unusable. With L1 unreachable, startup must +// return the retryable `L1ViewStale` verdict (exit 20) without mutating the +// batch tree. Restoring both time and connectivity must make the next boot +// healthy. Sub-block regressions remain tolerated and are pinned by protocol +// unit tests. +async fn run_wall_clock_backward_jump_retries_then_recovers_test( runtime: &mut ManagedSequencer, ) -> ScenarioResult<()> { let alice = TestSigner::from_default(1)?; @@ -1727,7 +2002,7 @@ async fn run_wall_clock_backward_jump_no_panic_test( let mut ws = runtime.ws(0).await?; let mut replay_before = ReplayWalletApp::devnet(); - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, @@ -1737,22 +2012,46 @@ async fn run_wall_clock_backward_jump_no_panic_test( .await?; drop(ws); + let counts_before = runtime.count_batches()?; runtime.stop().await?; let proxy = TcpProxy::spawn(runtime.l1_endpoint()).await?; runtime.set_l1_endpoint_override(Some(proxy.endpoint())); proxy.disconnect(); runtime.set_faketime_offset(Some("-1h".to_string()))?; - // Respawn must NOT panic. With L1 unreachable, the wall-clock fallback - // is the only path that sees `now - last_sync_ms` — if the subtraction - // ever became non-saturating, this call would panic via u64 underflow. - runtime.respawn().await?; + let respawn_error = runtime + .respawn() + .await + .expect_err("a one-hour clock regression with L1 unreachable must refuse admission"); + let respawn_error = respawn_error.to_string(); + assert!( + respawn_error.contains("status=exit status: 20"), + "backward-clock refusal must use the retryable exit class, not panic: {respawn_error}", + ); - // Clean up: reconnect and let the sequencer catch up normally. - proxy.reconnect(); - // Clear the offset for subsequent respawns (not used here, but keeps the - // teardown deterministic if future cleanup code respawns). + let counts_after_retry = runtime.count_batches()?; + assert_eq!( + counts_after_retry, counts_before, + "retryable clock refusal must not mutate the batch tree", + ); + + // Restore the usable clock and L1 connection. The same persisted facts + // are now ageable again, so startup admits and the runtime remains healthy + // past the danger detector's two-second cadence. runtime.set_faketime_offset(None)?; + proxy.reconnect(); + runtime.respawn().await?; + + let exit = runtime.observe_for(Duration::from_secs(3)).await?; + assert!( + exit.is_none(), + "restoring time and L1 connectivity must produce a stable runtime; got {exit:?}", + ); + assert_eq!( + runtime.count_batches()?, + counts_before, + "recovering from a retryable clock refusal must not invalidate batches", + ); proxy.shutdown().await?; Ok(()) @@ -1798,7 +2097,7 @@ async fn run_stalled_safe_head_startup_refuses_boot_test( let mut replay = ReplayWalletApp::devnet(); { let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay, @@ -1884,8 +2183,14 @@ async fn run_provider_outage_pre_danger_sequencer_continues_test( let mut replay = ReplayWalletApp::devnet(); { let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount) - .await?; + apply_reconciled_supported_deposit( + runtime, + &mut ws, + &mut replay, + &alice_l1, + deposit_amount, + ) + .await?; } // Step 2: Insert the proxy and route the sequencer through it via @@ -1999,7 +2304,7 @@ async fn run_provider_outage_danger_zone_sequencer_self_exits_test( let mut replay = ReplayWalletApp::devnet(); { let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay, @@ -2039,9 +2344,9 @@ async fn run_provider_outage_danger_zone_sequencer_self_exits_test( "sequencer must self-exit with non-zero status on danger detection, got {exit_status:?}", ); - // Step 5: Try to respawn while proxy is still disconnected. Startup - // runs the same wall-clock fallback via `run_preemptive_recovery` and - // should refuse to boot (`decide_startup_action → Refuse(...)`). + // Step 5: Try to respawn while proxy is still disconnected. The startup + // reducer runs the same wall-clock fallback and must return + // `Retry(L1ViewStale)` without runtime admission. let respawn_result = runtime.respawn().await; assert!( respawn_result.is_err(), @@ -2096,8 +2401,14 @@ async fn run_provider_outage_short_hiccup_no_recovery_test( let mut replay = ReplayWalletApp::devnet(); { let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit(runtime, &mut ws, &mut replay, &alice_l1, deposit_amount) - .await?; + apply_reconciled_supported_deposit( + runtime, + &mut ws, + &mut replay, + &alice_l1, + deposit_amount, + ) + .await?; } // Route through the proxy (stop → override → respawn). @@ -2185,7 +2496,7 @@ async fn run_both_down_danger_zone_sequencer_first_refuses_boot_test( let mut replay_before = ReplayWalletApp::devnet(); { let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, @@ -2235,8 +2546,8 @@ async fn run_both_down_danger_zone_sequencer_first_refuses_boot_test( // Complement to (sequencer first): here L1 comes back before the // sequencer does. Once the sequencer restarts, startup recovery sees L1 // reachable and the Tip aged past `danger_threshold`, so `check_danger` -// returns `TipInDanger(idx)` → `decide_startup_action` returns `RecoverTip` → -// `recover_aging_tip` cascades the Tip and opens a fresh one. Convergence +// returns `TipInDanger(idx)` and the guarded `RecoverTip` phase cascades the +// Tip and opens a fresh one. Convergence // typically happens on the first respawn. // // Other paths can fire under different timings — e.g., the lane might @@ -2272,7 +2583,7 @@ async fn run_both_down_danger_zone_proxy_first_restart_cycle_recovers_test( let mut replay_before = ReplayWalletApp::devnet(); { let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, @@ -2363,8 +2674,8 @@ async fn run_both_down_danger_zone_proxy_first_restart_cycle_recovers_test( // Other paths can fire depending on timing — e.g., the lane might close the // Tip into a nonced batch before the detector trips, the submitter might // get the batch onto L1 fresh, and convergence happens by the next respawn -// seeing it in `safe_inputs`. Or the closed batch lands stale, routes -// through `FlushAndCascade`, and converges after a flush cycle. +// seeing it in `safe_inputs`. Or the closed batch lands stale, routes through +// the guarded Flush → Sync → Cascade phases, and converges after a flush cycle. // // The test's load-bearing assertion is restart-loop convergence under a // realistic coupled outage, not which specific recovery path fires nor how @@ -2385,7 +2696,7 @@ async fn run_sequencer_outage_danger_zone_coupled_restart_cycle_recovers_test( let mut replay_before = ReplayWalletApp::devnet(); { let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, @@ -2418,7 +2729,7 @@ async fn run_sequencer_outage_danger_zone_coupled_restart_cycle_recovers_test( // Convergence is the load-bearing claim. The number of attempts depends // on which recovery path fires (`RecoverTip` typically converges on the - // first respawn; `FlushAndCascade` may take more), so we don't pin a + // first respawn; Flush → Sync → Cascade may take more), so we don't pin a // minimum here. assert!( !outcomes.is_empty(), @@ -2493,7 +2804,7 @@ async fn run_provider_outage_danger_zone_mid_run_exit_then_restart_cycle_recover let mut replay_before = ReplayWalletApp::devnet(); { let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, @@ -2586,9 +2897,9 @@ async fn run_provider_outage_danger_zone_mid_run_exit_then_restart_cycle_recover // first boot (needs L1 reachable to deploy contracts and pin the deployment // identity). We stop, rewrite the recorded L1 safe-head observation to // unknown, then respawn with the proxy disconnected. The deployment identity -// is still populated — so the sequencer gets past the contract-discovery -// phase — but `check_danger` sees the missing safe-head row and -// `decide_startup_action` returns `Refuse(L1ViewStale)`. +// is still populated — so the sequencer gets past the identity gate — but +// `check_danger` sees the missing safe-head row and the reducer returns +// `Retry(L1ViewStale)`. // // Scope note: a "truly" first-ever boot would fail even earlier (no // deployment identity, can't discover contracts). That's a separate test; this @@ -2673,7 +2984,7 @@ async fn run_delayed_inclusion_cascades_on_restart_test( let mut replay_before = ReplayWalletApp::devnet(); let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, @@ -2787,15 +3098,17 @@ async fn run_delayed_inclusion_cascades_on_restart_test( // // Staging: // 1. Baseline: deposit + transfer → Tip at first_frame_safe_block X. -// 2. `mine_l1_blocks(DANGER_ZONE_BLOCKS)` — current_safe_block jumps -// ~1150 past X, so the Tip's age clears `danger_threshold`. Wall -// clock untouched (decoupled advance). +// 2. Advance only L1 by `DANGER_ZONE_BLOCKS` — current_safe_block jumps +// ~1150 past X before the wall-clock batch deadline can close the Tip. +// Once the reader persists that head, observed Tip age outranks the +// deliberately future-dated L1 clock and clears `danger_threshold`. // 3. `wait_for_exit` — input reader catches up; detector ticks; sees // `DangerStatus::TipInDanger(_)`; process exits non-zero. -// 4. Respawn — startup `check_danger` again sees Tip in danger → +// 4. Align wall time with the already-mined L1 time, then respawn. Startup +// can now age the persisted head normally and sees Tip in danger → // `RecoverTip` → `recover_aging_tip` cascades the Tip + opens a fresh -// one. Alice's pre-outage transfer was a soft confirmation against -// the cascaded Tip; it's rolled back. +// one. Alice's pre-outage transfer was a soft confirmation against the +// cascaded Tip; it's rolled back. async fn run_aging_open_tip_runtime_danger_zone_exit_test( runtime: &mut ManagedSequencer, ) -> ScenarioResult<()> { @@ -2809,7 +3122,7 @@ async fn run_aging_open_tip_runtime_danger_zone_exit_test( let mut replay = ReplayWalletApp::devnet(); { let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay, @@ -2821,17 +3134,26 @@ async fn run_aging_open_tip_runtime_danger_zone_exit_test( replay.apply(ws.expect_user_op_from(alice_address).await?)?; } - // L1 jumps into the danger window; wall clock stays put. + // Advance L1 atomically while leaving wall time still: this injects the + // "lane failed to close its Tip" condition without letting the normal + // wall-clock batch deadline repair it first. runtime.mine_l1_blocks(DANGER_ZONE_BLOCKS).await?; // The detector must trip on `DangerStatus::TipInDanger` once the input reader // catches up. Allow a window for input-reader poll (~2 s) plus // detector poll (2 s) plus margin. let exit = runtime.wait_for_exit(Duration::from_secs(15)).await?; - assert!( - !exit.success(), - "sequencer must exit non-zero on `DangerStatus::TipInDanger` once the Tip's \ - first frame ages past `danger_threshold`, got {exit:?}", + // Exit 10 (`EXIT_RESTART_EXPECT_RECOVERY`) is the observed-danger class + // (`TipInDanger` / `ClosedBatchInDanger`); a clock fallback (`L1ViewStale`, + // `EstimatedBatchInDanger`) would exit 20. The scenario has no closed + // batch, so 10 here means the detector saw the aging Tip, not the + // future-dated L1 clock. The exit code is the contract; the rendered log + // line is not. + assert_eq!( + exit.code(), + Some(10), + "runtime detector must exit with the observed-danger class on \ + `DangerStatus::TipInDanger`, not a clock fallback; got {exit:?}", ); // No cascade fires on detector exit alone. The recovery happens at @@ -2844,6 +3166,14 @@ async fn run_aging_open_tip_runtime_danger_zone_exit_test( "detector exit alone must not invalidate batches; that happens at startup: {counts_before:?}", ); + // The reader's persisted L1 head is future-dated because Anvil advanced + // atomically. Add ordinary live progress, then align process time before + // restart: initial sync observes a strictly newer safe head and refreshes + // the wall-progress witness, so the mandatory post-repair inspection can + // admit the fresh Tip rather than correctly retaining a stale-view retry. + runtime.mine_live_l1_blocks(1).await?; + let aligned_offset_secs = blocks_as_duration(DANGER_ZONE_BLOCKS).as_secs() + 1; + runtime.set_faketime_offset(Some(format!("+{aligned_offset_secs}s")))?; runtime.respawn().await?; let mut ws_after = runtime.ws(0).await?; @@ -2908,7 +3238,7 @@ async fn run_stalled_safe_head_live_exit_test( let mut replay = ReplayWalletApp::devnet(); { let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay, @@ -2973,7 +3303,7 @@ async fn run_ws_reconnect_at_invalidated_offset_skips_cleanly_test( // Build up offsets 0 (deposit) and 1 (transfer) and capture the // transfer's offset so we can later reconnect at it. let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, @@ -3069,7 +3399,7 @@ async fn run_ws_subscribe_from_future_offset_waits_silently_test( let mut replay = ReplayWalletApp::devnet(); { let mut ws = runtime.ws(0).await?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay, @@ -3242,7 +3572,7 @@ async fn run_replay_matches_live_for_mixed_workload_test( // Diverse workload — exercises deposit-interleaving and every op // combination supported by the wallet app. - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_live, @@ -3255,7 +3585,7 @@ async fn run_replay_matches_live_for_mixed_workload_test( .await?; replay_live.apply(ws.expect_user_op_from(alice_address).await?)?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_live, @@ -3365,7 +3695,7 @@ async fn run_provider_outage_input_reader_retries_after_reconnect_test( // Baseline deposit with the proxy connected — proves the WS + reader // path works end-to-end before we break it. - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay, @@ -3419,9 +3749,9 @@ async fn run_provider_outage_input_reader_retries_after_reconnect_test( // recovery logic runs. // // Distinct from `run_first_boot_l1_unreachable_never_synced_refuses_boot_test` -// (already covered): that test exercises the wall-clock fallback inside -// `run_preemptive_recovery`, which only fires AFTER bootstrap discovery has -// succeeded once (so the deployment identity is pinned). This test targets +// (already covered): that test exercises the run reducer's wall-clock fallback, +// which only runs after setup has succeeded once (so deployment identity is +// pinned). This test targets // the earlier failure: the // `InputReader::new` discovery step where the sequencer asks L1 for the // InputBox address. With no deployment identity, that call has no @@ -3567,7 +3897,7 @@ async fn run_nonce_zero_recovery_invalidates_then_accepts_at_nonce_zero_test( { let mut ws = runtime.ws(0).await?; let mut alice_l2 = runtime.wallet_l2(alice.clone())?; - apply_safe_supported_deposit( + apply_reconciled_supported_deposit( runtime, &mut ws, &mut replay_before, diff --git a/tests/e2e/src/watchdog_compare.rs b/tests/e2e/src/watchdog_compare.rs index 249684a3..4ab77b0b 100644 --- a/tests/e2e/src/watchdog_compare.rs +++ b/tests/e2e/src/watchdog_compare.rs @@ -164,7 +164,7 @@ pub async fn run_watchdog_non_genesis_compare_test( } let decoded = wallet_snapshot::decode(body.as_slice()) .map_err(|err| format!("decode non-genesis finalized_state: {err}"))?; - if decoded.executed_input_count() == 0 { + if decoded.executed_input_count() == sequencer_core::history::ExecutedInputCount::ZERO { return Err("expected non-genesis finalized_state executed_input_count > 0".into()); } diff --git a/tests/harness/src/replay.rs b/tests/harness/src/replay.rs index ed88ff50..723a31bd 100644 --- a/tests/harness/src/replay.rs +++ b/tests/harness/src/replay.rs @@ -4,7 +4,7 @@ use alloy_primitives::{Address, U256}; use app_core::application::{WalletApp, WalletConfig}; use sequencer_core::api::WsTxMessage; -use sequencer_core::application::Application; +use sequencer_core::application::{Application, execute_direct_input, execute_valid_user_op}; use sequencer_core::l2_tx::{DirectInput, ValidUserOp}; use crate::HarnessResult; @@ -43,7 +43,7 @@ impl ReplayWalletApp { } pub fn executed_input_count(&self) -> u64 { - self.app.executed_input_count() + self.app.executed_input_count().get() } pub fn last_executed_safe_block(&self) -> u64 { @@ -62,11 +62,14 @@ pub(crate) fn apply_ws_message( payload, .. } => { - app.execute_direct_input(&DirectInput { - sender: decode_address(sender.as_str()), - block_number, - payload: decode_hex_prefixed(payload.as_str()), - })?; + execute_direct_input( + app, + &DirectInput { + sender: decode_address(sender.as_str()), + block_number, + payload: decode_hex_prefixed(payload.as_str()), + }, + )?; } WsTxMessage::UserOp { sender, @@ -75,7 +78,8 @@ pub(crate) fn apply_ws_message( safe_block, .. } => { - app.execute_valid_user_op( + execute_valid_user_op( + app, &ValidUserOp { sender: decode_address(sender.as_str()), fee, diff --git a/tests/harness/src/rollups.rs b/tests/harness/src/rollups.rs index e1f53bb8..395e6845 100644 --- a/tests/harness/src/rollups.rs +++ b/tests/harness/src/rollups.rs @@ -9,7 +9,7 @@ use std::time::Duration; use alloy::network::{EthereumWallet, TransactionBuilder}; use alloy::providers::ext::AnvilApi; use alloy::providers::{Provider, ProviderBuilder}; -use alloy::rpc::types::TransactionRequest; +use alloy::rpc::types::{BlockNumberOrTag, TransactionRequest}; use alloy::signers::local::PrivateKeySigner; use alloy::sol_types::SolCall; use alloy_primitives::{Address, B256, Bytes, U256}; @@ -31,6 +31,7 @@ pub const DEVNET_CHAIN_ID: u64 = 31_337; const DEFAULT_ANVIL_START_TIMEOUT: Duration = Duration::from_secs(10); const DEFAULT_ANVIL_SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(3); const DEFAULT_ANVIL_SLOTS_IN_EPOCH: u64 = 1; +const LIVE_L1_BLOCK_INTERVAL_SECONDS: u64 = 1; const DEVNET_MOCK_ERC20_DEPLOYER_PRIVATE_KEY: &str = "0x59c6995e998f97a5a0044976f1d86dbce6c5bb4f80a8b5148f7f4f6d0d0c0abc"; const DEVNET_MOCK_ERC20_DEPLOYER_FUNDING_WEI: u64 = 1_000_000_000_000_000; @@ -87,10 +88,35 @@ impl DevnetRollupsStack { deploy_mock_erc20_from_default_funder(self.anvil.endpoint.as_str()).await } + /// Mine outage-style L1 progress at the configured 12-second block time. + /// Tests that model ordinary live progress should use + /// [`Self::mine_live_l1_blocks`] instead. pub async fn mine_l1_blocks(&self, block_count: u64) -> HarnessResult<()> { self.anvil.mine_blocks(block_count).await } + /// Mine ordinary live-chain progress without simulating a 12-second outage + /// per block. The one-second interval keeps Anvil timestamps monotone while + /// wall-clock-paced polling avoids tripping the clock-usability guard. + pub async fn mine_live_l1_blocks(&self, block_count: u64) -> HarnessResult<()> { + self.anvil + .mine_blocks_with_interval(block_count, LIVE_L1_BLOCK_INTERVAL_SECONDS) + .await + } + + pub async fn l1_safe_block_number(&self) -> HarnessResult { + let provider = ProviderBuilder::new() + .connect(self.anvil.endpoint.as_str()) + .await + .map_err(|err| io_other(format!("failed to connect anvil provider: {err}")))?; + let block = provider + .get_block_by_number(BlockNumberOrTag::Safe) + .await + .map_err(|err| io_other(format!("failed to read Anvil safe head: {err}")))? + .ok_or_else(|| io_other("Anvil returned no safe block"))?; + Ok(block.header.number) + } + /// Toggle Anvil's auto-mining mode. When disabled, txs accumulate in /// the mempool until an explicit `anvil_mine` call (or re-enable). pub async fn set_automine(&self, enabled: bool) -> HarnessResult<()> { @@ -236,7 +262,15 @@ impl ManagedAnvil { // spurious `L1ViewStale` even when wall clock and L1 should move // together. See `ManagedSequencer::advance_wall_and_mine`. const SECONDS_PER_BLOCK: u64 = 12; + self.mine_blocks_with_interval(block_count, SECONDS_PER_BLOCK) + .await + } + async fn mine_blocks_with_interval( + &self, + block_count: u64, + interval_seconds: u64, + ) -> HarnessResult<()> { if block_count == 0 { return Ok(()); } @@ -246,7 +280,7 @@ impl ManagedAnvil { .await .map_err(|err| io_other(format!("failed to connect anvil provider: {err}")))?; provider - .anvil_mine(Some(block_count), Some(SECONDS_PER_BLOCK)) + .anvil_mine(Some(block_count), Some(interval_seconds)) .await .map_err(|err| { io_other(format!( diff --git a/tests/harness/src/sequencer.rs b/tests/harness/src/sequencer.rs index 620d2f89..71d166cd 100644 --- a/tests/harness/src/sequencer.rs +++ b/tests/harness/src/sequencer.rs @@ -26,9 +26,10 @@ const DEFAULT_SEQUENCER_START_TIMEOUT: Duration = Duration::from_secs(10); const DEFAULT_SEQUENCER_SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(3); /// Cadence at which the harness mines an L1 block during a recovery boot's /// readiness wait when [`ManagedSequencer::set_mine_l1_during_boot`] is on. -/// Fast enough that Anvil's `safe` tag advances past the flushed nonce within -/// the readiness window; the exact value is a harness detail, not test-visible. -const BOOT_L1_MINE_INTERVAL: Duration = Duration::from_millis(200); +/// One block per wall-clock second keeps the ordinary live-chain timestamp +/// coupled while still advancing Anvil's `safe` tag well within the recovery +/// readiness window. +const BOOT_L1_MINE_INTERVAL: Duration = Duration::from_secs(1); /// Readiness budget for a *recovery* boot (mining-during-boot enabled). The WP2 /// mempool flush polls `get_transaction_count(Safe)` once per L1 block time /// (`safe_poll_interval = seconds_per_block`, 12 s here), so it cannot resolve @@ -439,6 +440,41 @@ impl ManagedSequencer { .map_err(|_| io_other(format!("first frame fee out of range: {fee}")).into()) } + /// Return `(open_frame_safe_block, persisted_l1_safe_head)` from one + /// read-only SQLite snapshot. This observes the two sides of the live + /// frame-clock condition without relying on incidental Anvil transaction + /// counts. + pub fn frame_clock_observation(&self) -> HarnessResult<(u64, u64)> { + let db_path = self.data_dir_path.join("sequencer.db"); + let conn = rusqlite::Connection::open_with_flags( + db_path.as_path(), + rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY, + ) + .map_err(|err| io_other(format!("open DB read-only: {err}")))?; + let (frame_safe_block, persisted_safe_head): (i64, i64) = conn + .query_row( + "SELECT f.safe_block, h.block_number \ + FROM valid_open_batch b \ + JOIN frames f ON f.batch_index = b.batch_index \ + JOIN l1_safe_head h ON h.singleton_id = 0 \ + ORDER BY f.frame_in_batch DESC LIMIT 1", + [], + |row| Ok((row.get(0)?, row.get(1)?)), + ) + .map_err(|err| io_other(format!("read frame-clock observation: {err}")))?; + let frame_safe_block = u64::try_from(frame_safe_block).map_err(|_| { + io_other(format!( + "open-frame safe block is negative: {frame_safe_block}" + )) + })?; + let persisted_safe_head = u64::try_from(persisted_safe_head).map_err(|_| { + io_other(format!( + "persisted L1 safe head is negative: {persisted_safe_head}" + )) + })?; + Ok((frame_safe_block, persisted_safe_head)) + } + /// Copy the current finalized snapshot dump to `/checkpoint` (which /// survives [`Self::reset_database`], since that only clears `sequencer.db*` /// and `dumps/`), returning the captured checkpoint. Call after a batch has @@ -782,10 +818,22 @@ impl ManagedSequencer { self.rollups.deploy_extra_mock_erc20().await } + /// Mine outage-style L1 progress at 12 seconds per block. Pair large + /// advances with faketime, or use [`Self::mine_live_l1_blocks`] for + /// ordinary liveness and finality polling. pub async fn mine_l1_blocks(&self, block_count: u64) -> HarnessResult<()> { self.rollups.mine_l1_blocks(block_count).await } + /// Mine ordinary live-chain progress at one second per block. + pub async fn mine_live_l1_blocks(&self, block_count: u64) -> HarnessResult<()> { + self.rollups.mine_live_l1_blocks(block_count).await + } + + pub async fn l1_safe_block_number(&self) -> HarnessResult { + self.rollups.l1_safe_block_number().await + } + /// Toggle Anvil's auto-mining mode. When disabled, txs accumulate in /// the mempool until an explicit mine or re-enable. Used to hold a /// sequencer's batch-submission tx out of a block while the chain @@ -1094,7 +1142,11 @@ impl ManagedSequencer { Err(_) => { self.child.start_kill()?; let _ = self.child.wait().await; - Ok(()) + Err(io_other(format!( + "sequencer did not drain within {:?}; forced kill was required", + self.shutdown_timeout + )) + .into()) } } } @@ -1281,7 +1333,7 @@ async fn spawn_sequencer_process( )) .into()); } - let _ = rollups.mine_l1_blocks(1).await; + let _ = rollups.mine_live_l1_blocks(1).await; tokio::time::sleep(BOOT_L1_MINE_INTERVAL).await; } } else { @@ -1363,7 +1415,7 @@ async fn spawn_sequencer_process( // Transient RPC hiccups mid-boot are non-fatal — the next tick // retries. A genuine mining failure manifests as the boot // timing out, reported by the readiness arm. - let _ = rollups.mine_l1_blocks(1).await; + let _ = rollups.mine_live_l1_blocks(1).await; } }; tokio::select! { diff --git a/tests/harness/src/wallet.rs b/tests/harness/src/wallet.rs index a825e81c..56ef5cdd 100644 --- a/tests/harness/src/wallet.rs +++ b/tests/harness/src/wallet.rs @@ -156,17 +156,17 @@ impl WalletL1Client { &self, token_address: Address, amount: U256, - ) -> HarnessResult<()> { + ) -> HarnessResult { self.mint_token(token_address, amount).await?; self.deposit_token(token_address, amount).await } - pub async fn mint_and_deposit_supported_token(&self, amount: U256) -> HarnessResult<()> { + pub async fn mint_and_deposit_supported_token(&self, amount: U256) -> HarnessResult { self.mint_supported_token(amount).await?; self.deposit_supported_token(amount).await } - pub async fn deposit_token(&self, token_address: Address, amount: U256) -> HarnessResult<()> { + pub async fn deposit_token(&self, token_address: Address, amount: U256) -> HarnessResult { let token = MockERC20::new(token_address, &self.provider); let approve_receipt = token .approve(self.erc20_portal_address, amount) @@ -208,10 +208,13 @@ impl WalletL1Client { "failed to confirm ERC20 portal deposit transaction: {err}" )) })?; - ensure_success(deposit_receipt.status(), "ERC20 portal deposit") + ensure_success(deposit_receipt.status(), "ERC20 portal deposit")?; + deposit_receipt + .block_number + .ok_or_else(|| io_other("ERC20 portal deposit receipt has no block number").into()) } - pub async fn deposit_supported_token(&self, amount: U256) -> HarnessResult<()> { + pub async fn deposit_supported_token(&self, amount: U256) -> HarnessResult { self.deposit_token(self.supported_erc20_token, amount).await } }