fix(wallet): replace a contradicted writer instead of discarding the quorum - #222
Conversation
aff9dad to
cd4460a
Compare
There was a problem hiding this comment.
Verdict: CHANGES-REQUIRED (comment review; request-changes is a 422 on an own-authored PR)
Correction: an earlier revision of this review body was posted with the WRONG CONTENT — a sibling lane's findings about
src/peer/mod.rs, caused by a filename collision in the sharedC:/tmp. That text never applied to this PR. The body below is the real review of #222 atcd4460a1.
CHANGES-REQUIRED — one gating finding, head cd4460a1
Read-only gate from an independent clone. cargo test -p dig-wallet --lib = 534 passed / 0 failed. Revert-probe: replacing both replica_answer_is_current(...) calls with the old synced: true makes a_behind_replica_serves_its_figure_and_says_it_is_not_current and the_coin_read_reports_the_same_freshness_as_the_balance_read FAIL — the tests genuinely discriminate.
Verified good: may_elevate is byte-identical (diffed the extracted fn body). refusal_agrees_with_may_elevate_on_every_input is exhaustive, not sampled: Verdict has exactly 4 variants, every_verdict() lists all 4, crossed with agree/contradict/silent, and neither function reads height. Undecided is tested first; WriterSilent covers Ok(None) and the folded Err and is excluded from counts_as_disagreement(). an_incomplete_catch_up_never_serves_a_dated_zero_from_the_replica asserts the peak is PRESENT in the fixture, so it does hold the db_synced axis — the argument for keeping that axis is sound, since peak_height advances from new_peak_wallet independently of coverage.
The one blocker is below: is_following fails OPEN on an unobservable peer tier, so the balance path still presents a stale figure as current — measured, not reasoned.
Stub commit so the branch and its draft PR exist before implementation. Co-Authored-By: Claude <noreply@anthropic.com>
…quorum A decisive quorum that the writer contradicts identifies the WRITER as the anomaly, but the supervisor folded that round into the same refusal as a split and a probe failure: it kept the writer and held the session for the full `RECORROBORATE_AFTER` interval before dialling anyone else. The replica went unwritten for that whole time, and the round was counted as peers disagreeing even when the only thing that had happened was a slow peer failing to answer. `may_elevate` is left byte-identical. It is the write gate, it was already correct, and leaving it untouched is how "a contradicting writer still may not write" stays true by construction rather than by review. The new `refusal` classifier sits beside it and only names WHICH party the round accused; `refusal_agrees_with_may_elevate_on_every_input` pins the two together over the whole input space so the richer function cannot become a second door. The quorum's answer never becomes a write. It is settled at a deliberately lagged `common_height`, and acting on it would put a chain fact into `sync_state` with no `WriteAuthority` holder — the exact bypass corroboration exists to prevent. Only the writer's fate changes. Three refusals, three responses: * Undecided (split, insufficient, no corroborator, probe failure) — unchanged in every respect, including both log lines. * WriterContradicted — counted toward `PERSISTENT_DISAGREEMENT_ROUNDS` so the partition warning still escalates, logged naming the writer, and the session ends at once. * WriterSilent — NOT counted. Silence is not a contradiction, and spending a slow peer as evidence walks the node toward a partition warning it has no evidence for. The replacement introduces no new constant: a refused session is far shorter than `HEALTHY_SESSION`, so backoff is not reset and a permanent, locally-caused mismatch converges on one dial per `BACKOFF_MAX`. Refs dig_ecosystem#2868 Co-Authored-By: Claude <noreply@anthropic.com>
…rrent The `Source::Db` arms of `balance_for_address` and `coins_for_address` set `synced` to the literal `true`, so every replica-served answer claimed to be at the tip. That claim was never checked against anything: the flag that selects the tier is `initial_sync_complete`, which LATCHES — it records that a catch-up once finished, and only a backwards chain move clears it — so a replica hundreds of blocks behind still routes to the replica and still reported its figure as current. `synced` is now measured, from the same `is_following` predicate `control.wallet.syncStatus` reports its phase from, so the two endpoints cannot disagree about the same moment. A behind replica keeps SERVING, and keeps reporting its real `peak_height`: `synced: false` beside a height means "this figure is real, as of that height", which is a usable answer, not a withheld one. Fallback answers are unchanged — `false` / `null`. The `db_synced` axis of `routing::route` is deliberately NOT removed. Measured on the live node: `peak_height = 9140640`, `initial_sync_complete = 0`, and zero coin rows. `peak_height` advances from `new_peak_wallet` independently of any coin being applied, so a present peak is evidence about the chain and never about this replica's coverage. Serving that state would render as "Balance: 0, correct as of block 9,140,640" for a wallet holding 1.599 XCH. `an_incomplete_catch_up_never_serves_a_dated_zero_from_the_replica` holds the axis in place with a fixture built in exactly that state. Version coupling: an un-upgraded dig-app refuses any `!synced` reading, so it will stop showing balances it shows today. The dig-app half of #2869 lands first. Refs dig_ecosystem#2869 Co-Authored-By: Claude <noreply@anthropic.com>
SPEC described the refusal as one outcome ("no corroborated answer") and
`synced` as a property a `"db"` answer simply has. Both are now false of the
implementation, and a spec left describing the two-way fold is the version a
reimplementation would build.
Adds the three-refusal table (undecided / writer contradicted / writer silent),
the load-bearing arm order, the prohibition on adopting the quorum's answer as
a write, and the requirement that the replacement be bounded by the existing
reconnect ladder rather than a new constant. §18.7b now states that `synced` is
MEASURED, that a `"db"` answer with `synced: false` beside a `peak_height` is a
real figure that MUST still be served, and why the latching
`initial_sync_complete` cannot answer the question.
The acceptance script's gate 3 stopped requiring the replica to be within 50
blocks of the tip: a behind replica must still serve, so failing the run on the
distance would fail the correct behaviour. It now requires the replica to have a
height to answer as of, reports the distance as context, and asserts the
falsehood that was actually removed — a far-behind replica may serve, but may
never report `synced: true`.
Refs dig_ecosystem#2868, dig_ecosystem#2869
Co-Authored-By: Claude <noreply@anthropic.com>
Minor: `synced` on a `Source::Db` answer becomes a measured value, and the supervisor gains a public `RefusalReason` / `refusal` / `SessionTrust` surface. No API is removed and no wire field is added or removed, so this is compatible new capability rather than a break — but the MEANING of an existing `synced: true` narrows, and a consumer that requires it is affected, which is why it is not a patch. Refs dig_ecosystem#2868 Co-Authored-By: Claude <noreply@anthropic.com>
cd4460a to
10b0238
Compare
`is_following` answers `true` on an unobservable peer tier by design: on `control.wallet.syncStatus` an absent second opinion is not an accusation. A money read delegating to it unnarrowed therefore paired `synced: true` with an arbitrarily old `peak_height` whenever no chain peer had announced a height — a freshly started node, or one with no reachable chain peer — resting that claim on the latched `initial_sync_complete` this change exists to stop trusting. `replica_answer_is_current` now requires an observable tier before delegating. `is_following` itself is untouched, so the two endpoints keep agreeing wherever a peer height exists. The figure is still SERVED with its real `peak_height`, labelled stale. Also re-anchors `an_incomplete_catch_up_never_serves_a_dated_zero_from_the_replica` to `replica_is_authoritative` — dig_ecosystem#2871 replaced `is_synced` at both production call sites feeding `route`, so the test had drifted off the predicate it describes — and couples the acceptance script's stale assertion to the unobservable-peer state its `behind != unknown` guard cannot see. Refs dig_ecosystem#2869 Co-Authored-By: Claude <noreply@anthropic.com>
…eight Co-Authored-By: Claude <noreply@anthropic.com>
|
Resume-ready progress — branch DONE:
EVIDENCE: NEXT ACTION: watch |
MichaelTaylor3d
left a comment
There was a problem hiding this comment.
Verdict: PASS — scoped correctness re-gate
Head read: 5844a55a23cdec1175968e2c162040d72482112f (resolved from gh pr view --json headRefOid; unchanged during the review). Read-only gate from my OWN detached worktree cut from the dig-node checkout; no shared checkout was mutated (my worktree was verified clean and removed).
Scope was the narrow correctness leg only (crypto/custody untouched by this delta; loop-security and the adversarial leg deliberately not run). §2.6 phase-1 posture applied: gating limited to red CI, a money/custody lie, or a destructive action beyond scope.
Measured, by execution
cargo test -p dig-wallet --lib= 546 passed / 0 failed / 1 ignored. Matches the lane's claim exactly (the gate's 534 oncd4460a1+ #223's tests + the new one).git diff origin/main..HEAD(origin/main=dcd1e9e) = 7 files, +958 / -76. The lane's comment says +854/-71 — that figure predates its own last two commits (d81bc45,5844a55). Stale evidence figure, not a code issue.- All required checks GREEN by name (
Test + coverage,Clippy,Rustfmt,Check version increment,Lint commit messages, CodeQL x3, the four native package builds).
The three prior findings — all genuinely closed, verified by mutation
- GATING finding closed.
crates/dig-wallet/src/sage/rpc.rs:850now returnsfalsewhenchain_peer_tier().peak_heightisNone, then delegates tois_followingunchanged. Probe (executed): replacing that body with the pre-fixis_following(peak_height, self.chain_peer_tier().await.peak_height)makesan_unobservable_peer_tier_is_never_reported_as_currentFAIL atrpc.rs:6052— "a figure no peer height could corroborate was reported as current". The test discriminates; it is not a false green. - Re-anchor verified.
an_incomplete_catch_up_never_serves_a_dated_zero_from_the_replicanow callsbe.replica_is_authoritative(). Probe (executed): forcingreplica_is_authoritativetoOk(true)makes it FAIL atrpc.rs:6162. Anchored to the predicate production actually consults. - Acceptance script now fails gate 4 on
behind == unknown && synced == True(scripts/acceptance-wallet-balance.sh:87-89), and additionally requirespeak_height != Noneon a db answer. The pair no longer fails open together.
The HARD CONSTRAINT held
is_following (sync_supervisor.rs:576-580) is byte-identical — the only diff hunk near it is the header line for may_elevate, which is also unchanged. crates/dig-wallet/src/sage/sync_supervisor/tests.rs has zero deleted lines (+333, pure addition). The narrowing lives at the replica_answer_is_current call site only, so control.wallet.syncStatus keeps its tolerant answer.
Fail-closed DIRECTION — asked both ways
- Does it ever claim currency nothing corroborated? Only in the residual case below (which is pre-existing, not introduced here). The peer-side hole is closed.
- Does it ever withhold money, or report
synced: falseforever on a healthy node? No. The balance and coin set are always still served with the realpeak_height;syncedis passed straight through to JSON indig-node-service/src/control.rs:2217,2239,2279,2296and gates no send/spend path anywhere in the node. ProductionChainTransport::peer_tier(chain.rs:151) reports the held client'speer_peak_height(), so a node with a genuinely reachable peer tier reachessynced: true— pinned bya_replica_level_with_its_peers_still_reports_synced. The worst case on an unobservable tier is the conservative direction (served figure, labelled stale), which is exactly #2869's premise, not a violation of it.
The five pre-existing tests — no weakening
Each hunk is a pure single-line addition of .with_chain_peer_tier_for_tests(peers_level_at(500)); no assertion was removed, altered or relaxed in any of them:
| Test | Now passes because | Verdict |
|---|---|---|
scoped_synced_reads_db_separating_confirmed_pending_and_spent (rpc.rs:4562) |
replica peak Some(500) == tier peak Some(500) → genuinely level |
Strictly stronger than the old literal true |
a_db_served_read_reports_the_db_tier_and_the_replicas_peak (:4714) |
same | Strictly stronger |
a_synced_owned_address_reads_its_real_unspent_coins_from_the_replica (:5793) |
same | Strictly stronger |
an_enrolled_address_reads_from_the_replica_not_the_oracle (:4747) |
replica peak None → is_following's tolerant arm |
Equal to before (its subject is the TIER, asserted verbatim: "only a Db answer may report itself synced") |
synced_empty_address_is_zero_success_not_error (:5142) |
same tolerant arm | Equal to before |
No test passes for a new reason in a way that hides anything; the last two rows are the surface of the off-path finding below rather than a weakening.
Off-path, NON-GATING — for the orchestrator to ticket
1. The narrowing is one-sided: the REPLICA side of is_following still fails open, so a db answer can report synced: true with peak_height: null. (crates/dig-wallet/src/sage/rpc.rs:850)
replica_answer_is_current guards only the peer argument. With replica peak None and an observable peer tier, is_following(None, Some(peer)) hits the _ => true arm. Executed probe on this head:
PROBE1 source=Db synced=true peak=None balance=4242
That is a claim of currency resting solely on the latched initial_sync_complete, with no height at all attached — the same class as the finding this round closed, reached from the other argument. It is production-reachable: complete_catch_up (db.rs:607) sets the peak and the latch together, but the other writer — the live-broadcast point-read refresh at rpc.rs:2599 — calls set_initial_sync_complete(true) + record_coverage() and never writes a peak, over a sync_state row whose peak_height defaults to NULL (db.rs:239). replica_is_authoritative (rpc.rs:751) does not require a peak either.
Not gating: this state reported synced: true before this PR too (the arm was a literal true), so the change does not introduce the lie — it leaves one door of it open while closing the other. It is also partly mitigated at acceptance time by the new peak != None check in gate 4. SPEC §18.7b as written does not describe this state. Suggested fix is one line at the same call site (peak_height.is_none() → false) plus the SPEC sentence. Deliberately NOT handed to Copilot — it is a money-freshness predicate and the wrong fix (widening is_following, or blanking the answer) is plausible.
2. Doc/evidence nit: the lane's progress comment states +854/-71; the actual delta at this head is +958/-76. Harmless, but a stated figure that no longer matches its own head.
3. Noted and accepted, no ticket needed: gate 3 of the acceptance script no longer fails on replica distance. That is correct per #2869 (a behind replica must still serve), and the distance-lie is still caught by gate 4's behind > 50 && synced == True. Likewise, the immediate replacement of a contradicted writer is a denial shape whose "excludes everyone" end state is the replica going unwritten — bounded by the existing backoff ladder, stated in SPEC, and pinned by repeated_contradiction_climbs_the_existing_backoff.
Verified by execution: the test count, both mutation probes, and PROBE1. Verified by reading: the five fixture hunks, is_following/may_elevate immutability, the synced pass-through in control.rs, and the SPEC/acceptance diffs.
PR remains DRAFT — undrafting is the orchestrator's step.
DO NOT MERGE — gate round in progress.
Closes DIG-Network/dig_ecosystem#2868 and the node half of DIG-Network/dig_ecosystem#2869.
Four commits, one per concern: the supervisor fix, the
syncedhonesty fix, the docs/acceptance sweep, and the version bump.Part 1 — #2868: a contradicted writer is replaced, not kept
Two corrections to the ticket's premises, both measured
await_recorroborationalready ends a refused session afterRECORROBORATE_AFTER= 45s (sync_supervisor.rs:97, armed in the sessionselect!). The real loss was 45s per bad writer plus a mis-attributedsplitsincrement — not 600s.common_height = min(claimed peaks) − SETTLED_LAG(quorum.rs:569), a deliberately lagged height, andsync.rsalready refuses backwards peaks — so acting on it could never makepeak_heighttrack the tip. Worse, it would reachsync_statewith noWriteAuthorityholder, on a sample the code itself documents as biased (connect_random_peertries127.0.0.1first). A hostile quorum under the shipped shape costs liveness; under the rejected one it buys a written chain fact with nobody to blame.The change
may_elevateis byte-identical. It is the write gate, it was already correct, and leaving it untouched is how "a contradicting writer still may not write" stays true by construction rather than by review.Beside it,
refusal()names WHICH party the round accused —Undecided/WriterSilent/WriterContradicted— andSessionTrustcarries that verdict out oftrust_for_sessionalongside the authority.Undecidedis tested FIRST: without a verdict there is nothing for the writer to have contradicted, and getting that order wrong accuses an honest writer on every split, which is a re-dial loop rather than a mislabel.How the three constraints are held:
may_elevateunchanged; every refusal path returnsWriteAuthority::DiscoveredviaSessionTrust::refused.refusal_agrees_with_may_elevate_on_every_inputpins the two functions together over the whole input space, so the richer one cannot become a second door.WriterSilentcovers both routes (Ok(None)and theErrthe call site folds into it). It is not counted towardPERSISTENT_DISAGREEMENT_ROUNDSand waits the unchanged 45s.await_recorroborationis parameterised over the EXISTING ladder; a contradictedDiscoveredsession subscribes nothing (so ending it discards no work), exits asSessionOutcome::Endedwell underHEALTHY_SESSION, so backoff is not reset and doubles 1 to 2 to 4 to 8 up toBACKOFF_MAX. A locally-caused permanent mismatch converges on one dial per minute.Undecidedkeeps both existing log lines verbatim.WriterContradictedstill incrementssplits, so the partition warning still escalates; it gets its own line naming the writer, and the persistent warning suppresses the per-round line so an operator does not count one round twice.Known follow-up, deliberately NOT fixed here
connect_random_peertakes no exclusion argument, so the replacement dial may return the same contradicting address. Bounded by the backoff ladder today. Filed rather than fixed.Part 2 — #2869 node half:
syncedbecomes measuredThe genuine node-side defect was not the
db_syncedtier gate. It wasrpc.rshardcodingsynced: truein bothSource::Dbarms, so once a catch-up HAD completed and the replica fell behind, the node reported a stale figure as current.db_syncedisinitial_sync_complete, which LATCHES (db.rs:478;sync_supervisor.rs:452— "persistent … only a backwards chain move clears it"), so a replica 530 blocks behind still routes to"db".syncednow comes fromis_following(replica_peak, peers_peak)— the SAME predicatecontrol.wallet.syncStatusderives its phase from, so the two endpoints cannot disagree about the same moment. A behind replica keeps SERVING and keeps reporting its realpeak_height;synced: falsebeside a height means "this figure is real, as of that height", not "unknown". Applied to the balance read, its CAT/$DIG twin (same function,assetparam) andcoins_for_address. Fallback answers unchanged (false/null).The
db_syncedaxis was NOT removed, and here is why#2869's Scope asks to remove it. Measured on the live 0.117.0 service:
sync_state.peak_height = 9140640,initial_sync_complete = 0,coinstable: 0 rows total.peak_heightadvances fromnew_peak_walletindependently of any coin being applied, so a present peak is evidence about the CHAIN and never about this replica's coverage. Serving that state renders as "Balance: 0. Correct as of block 9,140,640." for a wallet holding 1.599 XCH — well-formed, precisely dated, and false.an_incomplete_catch_up_never_serves_a_dated_zero_from_the_replicaholds the axis in place with a fixture built in exactly that state (peak PRESENT, flag 0, coins empty); a fixture leaving the peak unset would pass against the wrong implementation.Version coupling: an un-upgraded dig-app refuses any
!result.syncedreading (dig-app-core/src/wallet/node.rs:108), so it will stop showing balances it shows today. The dig-app half of #2869 lands first, by design.Blast radius checked
gitnexus is not indexed in this fresh worktree, so the radius was taken by call-graph grep plus direct read and is stated as such (§2.0 bound 2 — the obligation is the analysis, not the tool).
may_elevate— unchanged. Callers:trust_for_sessionand the existing pure-function test. No risk.trust_for_session— one caller (Supervisor::run), signature widened toSessionTrust. Every early return audited: operator/pre-corroborated, corroborator absent, probe error, refusal, elevation.await_recorroboration— one call site, in the sessionselect!. Theretry: booltoOption<RefusalReason>change preserves previous behaviour for every input exceptWriterContradicted(corroborator-absent and probe-error paths map toUndecided, which is the old 45s wait).WalletBalanceResult.synced/WalletCoinsResult.synced— produced inbalance_for_addressandcoins_for_addressonly; consumed over the wire by dig-app (coupling stated above) and bydig-node-control-interface. No field added or removed.routing::route— deliberately untouched.chain_peak()(rpc.rs:1447) derives itssyncedfrom the latchingdb.is_synced(), so it carries the same overstatement class. It is not hardcoded, and a caller bounding a claimed confirmation may REFUSE on!synced, so changing it is a behaviour change beyond this ticket's radius. Filed as a follow-up.Sibling PRs
feat/2606-chia-peer-pool) touchessync_supervisor.rs, but only changes how the corroborator SOURCES its sample (pool instead of fresh dials) — the verdict, refusal and elevation logic it feeds is untouched, so this change is orthogonal and survives the pool.rpc.rsbut does not alterbalance_for_address's result construction or anysynced: true, so there was no shape to align to.Tests — and the wrong implementation each catches
refusal_agrees_with_may_elevate_on_every_inputNoneand the supervisor hands it the replica. Table over all four verdicts times agreeing/contradicting/silent.an_undecided_round_never_accuses_the_writera_writer_that_could_not_answer_is_not_a_liarwriter_answer != Some(agreed)as the contradiction test — true forNone, so every slow peer is reported as a liar AND counted toward the partition warning. Both silence routes covered.a_contradicted_writer_ends_the_session_at_once_and_writes_nothingcatch_upnever called) AND the 45s hold surviving. The second assertion is about a PLACEMENT: "the session ended" is satisfied identically by the old path, so the observable is which duration the clock was asked for.a_non_decisive_quorum_still_changes_nothinga_writer_whose_probe_failed_is_not_replaced_at_onceUndecided.persistent_contradiction_still_reaches_the_partition_warningrepeated_contradiction_climbs_the_existing_backoffa_behind_replica_serves_its_figure_and_says_it_is_not_currentsynced: true. Asserts the balance AND peak alongside, so a "fix" that withholds the reading or blanks the peak fails.a_replica_level_with_its_peers_still_reports_syncedfalsehardcoded), which would make an upgraded client distrust every reading.the_coin_read_reports_the_same_freshness_as_the_balance_reada_fallback_answer_still_claims_neither_freshness_nor_a_heightan_incomplete_catch_up_never_serves_a_dated_zero_from_the_replicadb_syncedaxis. Fixture is the live measured state: peak present, flag 0, coins empty.Red proven, not assumed. The classifier was temporarily stubbed back to the two-way fold on a committed tree; five of the new supervisor tests failed on their own assertions (not on compilation), including
rung 0 should be ~1s, got 45s. Restoring the implementation returns 72 passed / 0 failed in the supervisor module, and the whole crate at 534 passed / 0 failed.cargo fmtclean,cargo clippy -p dig-wallet --all-featuresclean.Fixture sizing is taken from the protocol's own numbers rather than invented:
PEERS_AHEAD_BY = 530is the distance measured on the live node, chosen becauseFOLLOWING_TOLERANCEis four blocks and a fixture just inside it would assert the tolerance instead of the behaviour.Docs and acceptance
SPEC.md§18.6d gains the three-refusal table, the load-bearing arm order, the prohibition on adopting the quorum's answer as a write, and the no-new-constant bound.SPEC.md§18.7b now statessyncedis MEASURED, that a"db"answer withsynced: falsebeside apeak_heightis a real figure that MUST still be served, and why the latching flag cannot answer the question. Thecontrol.wallet.balance/.coinsmethod rows are updated to match.scripts/acceptance-wallet-balance.shgate 3 no longer requires the replica to be within 50 blocks of the tip — under #2869 a behind replica must still serve, so failing the run on the distance would fail the correct behaviour. It now requires the replica to have a height to answer as of, reports the distance as CONTEXT, and asserts the falsehood that was actually removed: a far-behind replica may serve but may never reportsynced: true. Gate 4 (source: "db") is kept unchanged.Version
0.117.0to0.118.0,dig-wallet0.21.0to0.22.0. Minor: compatible new capability (a publicRefusalReason/refusal/SessionTrustsurface), no API removed and no wire field added or removed — but the MEANING of an existingsynced: truenarrows and a consumer that requires it is affected, which is why it is not a patch.