From 13dc8f81443a4861c137e9b438a3799b1cd9d96c Mon Sep 17 00:00:00 2001 From: Michael Taylor Date: Thu, 13 Aug 2026 15:06:57 -0700 Subject: [PATCH 1/7] chore(wallet): open the lane for the quorum-replaces-writer work Stub commit so the branch and its draft PR exist before implementation. Co-Authored-By: Claude --- .lane-2868.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 .lane-2868.md diff --git a/.lane-2868.md b/.lane-2868.md new file mode 100644 index 00000000..1ddb809f --- /dev/null +++ b/.lane-2868.md @@ -0,0 +1 @@ +# Lane: #2868 quorum replaces the writer + #2869 node half From bd0e242797ebaad7e6457c853559e58db9de46d7 Mon Sep 17 00:00:00 2001 From: Michael Taylor Date: Thu, 13 Aug 2026 15:27:10 -0700 Subject: [PATCH 2/7] fix(wallet): replace a contradicted writer instead of discarding the quorum MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A decisive quorum that the writer contradicts identifies the WRITER as the anomaly, but the supervisor folded that round into the same refusal as a split and a probe failure: it kept the writer and held the session for the full `RECORROBORATE_AFTER` interval before dialling anyone else. The replica went unwritten for that whole time, and the round was counted as peers disagreeing even when the only thing that had happened was a slow peer failing to answer. `may_elevate` is left byte-identical. It is the write gate, it was already correct, and leaving it untouched is how "a contradicting writer still may not write" stays true by construction rather than by review. The new `refusal` classifier sits beside it and only names WHICH party the round accused; `refusal_agrees_with_may_elevate_on_every_input` pins the two together over the whole input space so the richer function cannot become a second door. The quorum's answer never becomes a write. It is settled at a deliberately lagged `common_height`, and acting on it would put a chain fact into `sync_state` with no `WriteAuthority` holder — the exact bypass corroboration exists to prevent. Only the writer's fate changes. Three refusals, three responses: * Undecided (split, insufficient, no corroborator, probe failure) — unchanged in every respect, including both log lines. * WriterContradicted — counted toward `PERSISTENT_DISAGREEMENT_ROUNDS` so the partition warning still escalates, logged naming the writer, and the session ends at once. * WriterSilent — NOT counted. Silence is not a contradiction, and spending a slow peer as evidence walks the node toward a partition warning it has no evidence for. The replacement introduces no new constant: a refused session is far shorter than `HEALTHY_SESSION`, so backoff is not reset and a permanent, locally-caused mismatch converges on one dial per `BACKOFF_MAX`. Refs dig_ecosystem#2868 Co-Authored-By: Claude --- .lane-2868.md | 1 - crates/dig-wallet/src/sage/sync_supervisor.rs | 252 +++++++++++-- .../src/sage/sync_supervisor/tests.rs | 333 ++++++++++++++++++ 3 files changed, 547 insertions(+), 39 deletions(-) delete mode 100644 .lane-2868.md diff --git a/.lane-2868.md b/.lane-2868.md deleted file mode 100644 index 1ddb809f..00000000 --- a/.lane-2868.md +++ /dev/null @@ -1 +0,0 @@ -# Lane: #2868 quorum replaces the writer + #2869 node half diff --git a/crates/dig-wallet/src/sage/sync_supervisor.rs b/crates/dig-wallet/src/sage/sync_supervisor.rs index d71e13c5..ac308d94 100644 --- a/crates/dig-wallet/src/sage/sync_supervisor.rs +++ b/crates/dig-wallet/src/sage/sync_supervisor.rs @@ -864,6 +864,110 @@ pub fn may_elevate(round: &CorroborationRound, writer_answer: Option) - } } +/// WHY a round refused to elevate its writer — the three refusals [`may_elevate`] deliberately +/// does not distinguish (dig_ecosystem#2868). +/// +/// [`may_elevate`] answers a WRITE question and answers it closed: every one of these means +/// "nothing may be written". This answers a different, purely diagnostic question — *which party +/// is the anomaly* — because the two refusals point at opposite culprits and deserve opposite +/// responses. It grants nothing; a caller can only ever use it to decide how quickly to give up on +/// the peer, never to let the peer write. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum RefusalReason { + /// The QUORUM reached no answer — a [`Verdict::Split`] or a [`Verdict::Insufficient`]. + /// + /// The truth is unknown, so nothing whatsoever is known about the writer either. Accusing it + /// here would re-dial on every split, which is a dial loop dressed as a defence. + Undecided, + /// The quorum was decisive and the writer did not answer at all — a probe error, or an honest + /// `None`. + /// + /// Silence is not a contradiction. This is what a slow or busy peer looks like, and it is the + /// one refusal that is not evidence of anything. + WriterSilent, + /// The quorum was decisive and the writer answered something ELSE at the same settled height. + /// + /// This is the only refusal that names a culprit: past the lag filter, at a height the writer + /// did not choose, disagreeing with an independently drawn quorum is a lie, a partition, or a + /// fork — never ordinary lag. + WriterContradicted, +} + +impl RefusalReason { + /// Whether this refusal is EVIDENCE that peers disagree about settled chain state — the thing + /// `splits` counts and [`quorum::PERSISTENT_DISAGREEMENT_ROUNDS`] escalates on. + /// + /// [`Self::WriterSilent`] is the one that is not. A peer that did not answer has contradicted + /// nothing, and counting it would walk a node with one slow peer toward a partition warning it + /// has no evidence for — a diagnostic that cries wolf is worse than no diagnostic, because the + /// real thing then reads as more of the same. + pub fn counts_as_disagreement(self) -> bool { + match self { + Self::Undecided | Self::WriterContradicted => true, + Self::WriterSilent => false, + } + } +} + +/// Classify a refusal, or `None` when the round elevates its writer. +/// +/// Kept BESIDE [`may_elevate`] rather than replacing it, and deliberately not consulted by it: the +/// write gate stays one unchanged expression, so "a contradicting writer still may not write" +/// remains true by construction rather than by review of this richer function. +/// `refusal_agrees_with_may_elevate_on_every_input` pins the two together over the whole input +/// space. +/// +/// # The arm order is load-bearing +/// +/// [`RefusalReason::Undecided`] is tested FIRST. Without a decisive quorum there is nothing for the +/// writer to have contradicted, so an implementation that compared answers first would report a +/// perfectly honest writer as a liar on every split — and the caller replaces a contradicting +/// writer promptly, so that mistake is a re-dial on every split rather than a mere mislabel. +pub fn refusal(round: &CorroborationRound, writer_answer: Option) -> Option { + let Some(agreed) = round.verdict.corroborated() else { + return Some(RefusalReason::Undecided); + }; + match writer_answer { + None => Some(RefusalReason::WriterSilent), + Some(mine) if &mine != agreed => Some(RefusalReason::WriterContradicted), + Some(_) => None, + } +} + +/// What a session may write, together with WHY it may not — the pair +/// [`Supervisor::trust_for_session`] settles in one step. +/// +/// The two travel together because a refusal decides both the authority (always +/// [`sync::WriteAuthority::Discovered`]) and how long the session is worth holding, and a caller +/// that received only the first would have to re-derive the second from `is_authoritative()` — +/// which cannot tell the three refusals apart, and is exactly the fold this replaced. +#[derive(Debug, Clone, Copy)] +pub struct SessionTrust { + /// What this session may write. + pub authority: sync::WriteAuthority, + /// Why it was refused, or `None` when it was not. + pub refusal: Option, +} + +impl SessionTrust { + /// A session that may write: there is no refusal to explain. + fn elevated(authority: sync::WriteAuthority) -> Self { + Self { + authority, + refusal: None, + } + } + + /// A refused session. Every refusal path returns [`sync::WriteAuthority::Discovered`], which + /// writes nothing — the reason changes only how promptly the peer is replaced. + fn refused(reason: RefusalReason) -> Self { + Self { + authority: sync::WriteAuthority::Discovered, + refusal: Some(reason), + } + } +} + /// Evidence that the CHAIN advanced, drawn independently of the subscription session /// (dig_ecosystem#2851). /// @@ -1174,7 +1278,8 @@ impl Supervisor { // higher peak would inflate apparent confirmation counts (see [`sync::PeerTrust`] // for the inversion that made this the vulnerability). It runs as a write-free // session, which is what powers the live sync status. - let authority = self.trust_for_session(&*session, &mut splits).await; + let SessionTrust { authority, refusal } = + self.trust_for_session(&*session, &mut splits).await; let trust = authority.trust(); // Publish the trust the INSTANT it is settled, before a subscription set is resolved // for it — the order the supervisor genuinely learns the two facts. Until @@ -1292,7 +1397,10 @@ impl Supervisor { // decision back to the reconnect path, which draws an independent sample and runs // the `Corroborator` again. That is what makes the "re-drawing a fresh sample" log // line above TRUE; no new retry mechanism is introduced (dig_ecosystem#2827). - () = self.await_recorroboration(!trust.is_authoritative()) + // HOW LONG it is held depends on which party the round accused: a writer caught + // contradicting a decisive quorum is replaced at once, rather than kept for another + // `RECORROBORATE_AFTER` while the replica goes unwritten (dig_ecosystem#2868). + () = self.await_recorroboration(refusal) => SessionOutcome::Ended, // The peer went silent while the chain kept moving. `run` above is parked on a // `recv()` with no deadline, and for an AUTHORITATIVE subscribed session every @@ -1372,10 +1480,10 @@ impl Supervisor { &self, session: &dyn SyncSession, splits: &mut u32, - ) -> sync::WriteAuthority { + ) -> SessionTrust { let dialed = session.trust(); if dialed != PeerTrust::Discovered { - return match dialed { + return SessionTrust::elevated(match dialed { // The operator hand-configured this address, and corroboration only runs on the // discovery path below — so there is no independent anchor to build a ceiling // from, and inventing one would second-guess an explicit configuration. @@ -1383,17 +1491,20 @@ impl Supervisor { // Unreachable in practice: `trust()` reports the DIAL source, which is never // already-corroborated. Elevation is this function's own output. PeerTrust::Corroborated | PeerTrust::Discovered => sync::WriteAuthority::Discovered, - }; + }); } + // Corroboration switched off, and a probe that reached nobody, are both rounds that reached + // no answer — `Undecided`, which accuses no one and holds the session for the unchanged + // `RECORROBORATE_AFTER`. let Some(corroborator) = self.corroborator.as_ref() else { - return sync::WriteAuthority::Discovered; + return SessionTrust::refused(RefusalReason::Undecided); }; let round = match corroborator.corroborate().await { Ok(r) => r, Err(e) => { tracing::debug!(error = %e, "wallet sync: corroboration probe failed; the peer stays uncorroborated and writes nothing"); - return sync::WriteAuthority::Discovered; + return SessionTrust::refused(RefusalReason::Undecided); } }; @@ -1406,30 +1517,42 @@ impl Supervisor { } }; - if !may_elevate(&round, writer_answer) { - *splits = splits.saturating_add(1); - let persistent = *splits >= quorum::PERSISTENT_DISAGREEMENT_ROUNDS; - // A run of failures is not "the network is slow". A fresh random sample failing to - // agree, repeatedly, is what a partition and a sustained attack both look like from a - // light client, and retrying quietly forever would present both as a node that merely - // never finishes syncing. - if persistent { - tracing::warn!( - consecutive = *splits, - height = round.height, - peer = %session.peer_ip(), - verdict = ?round.verdict, - "wallet sync: peers persistently disagree about settled chain state; the replica is deliberately NOT being written. This is evidence of a network partition or a hostile peer set, not of a slow connection." - ); - } else { - tracing::info!( - consecutive = *splits, - height = round.height, - verdict = ?round.verdict, - "wallet sync: no corroborated answer this round; re-drawing a fresh sample" - ); + // The write gate is still `may_elevate` and nothing else; this only names the refusal. + if let Some(reason) = refusal(&round, writer_answer) { + if reason.counts_as_disagreement() { + *splits = splits.saturating_add(1); + } + // The escalation speaks for a STANDING condition and outranks the per-round lines + // below, which describe a single transient round. Emitting both would make an operator + // count one round twice. + if !Self::report_persistent_disagreement(&round, session, *splits) { + match reason { + RefusalReason::Undecided => tracing::info!( + consecutive = *splits, + height = round.height, + verdict = ?round.verdict, + "wallet sync: no corroborated answer this round; re-drawing a fresh sample" + ), + // Named separately from the line above because this round knows WHICH peer to + // be suspicious of, and "no corroborated answer" is not even true here: the + // quorum answered, and the writer is the one who did not match it. + RefusalReason::WriterContradicted => tracing::info!( + consecutive = *splits, + height = round.height, + peer = %session.peer_ip(), + verdict = ?round.verdict, + "wallet sync: the writer contradicted an independent quorum about settled \ + chain state; replacing the writer rather than the round" + ), + RefusalReason::WriterSilent => tracing::info!( + height = round.height, + peer = %session.peer_ip(), + "wallet sync: the writer could not answer the corroboration question; it \ + stays uncorroborated and writes nothing" + ), + } } - return sync::WriteAuthority::Discovered; + return SessionTrust::refused(reason); } *splits = 0; @@ -1452,12 +1575,39 @@ impl Supervisor { // never the constant: `PeakCeiling`'s own doc says a hardcoded ceiling would silently // become too tight if the lifetime moved UP, and reading the constant here is exactly the // hardcoding it warns about (dig_ecosystem#2851, F3). - sync::WriteAuthority::Corroborated(sync::PeakCeiling::from_corroborated( - round.height, - self.session_lifetime, + SessionTrust::elevated(sync::WriteAuthority::Corroborated( + sync::PeakCeiling::from_corroborated(round.height, self.session_lifetime), )) } + /// Surface a STANDING disagreement, and report whether it did. + /// + /// A run of refusals is not "the network is slow". Fresh random samples failing to agree, or a + /// writer contradicting them, repeatedly, is what a partition and a sustained attack both look + /// like from a light client, and retrying quietly forever would present both as a node that + /// merely never finishes syncing. + /// + /// Returns `true` when it warned, so each caller can emit its own — more specific, and + /// different per refusal — line only while the condition is still transient. Two lines about + /// one round would make an operator count one event twice. + fn report_persistent_disagreement( + round: &CorroborationRound, + session: &dyn SyncSession, + splits: u32, + ) -> bool { + if splits < quorum::PERSISTENT_DISAGREEMENT_ROUNDS { + return false; + } + tracing::warn!( + consecutive = splits, + height = round.height, + peer = %session.peer_ip(), + verdict = ?round.verdict, + "wallet sync: peers persistently disagree about settled chain state; the replica is deliberately NOT being written. This is evidence of a network partition or a hostile peer set, not of a slow connection." + ); + true + } + /// Resolve once the subscription set stops being empty. /// /// Returns a future that NEVER resolves when `poll` is false — a session that already @@ -1474,17 +1624,43 @@ impl Supervisor { } } - /// Resolve once a REFUSED session has been held long enough to be worth re-corroborating. + /// Resolve once a REFUSED session has been held long enough to be worth replacing. /// - /// Returns a future that NEVER resolves when `retry` is false. An authoritative session has + /// Returns a future that NEVER resolves for `None`, which is an ELEVATED session. It has /// nothing to re-corroborate — it already cleared the quorum — and ending it would discard a /// live subscription and force a fresh catch-up from genesis, which is a worse failure than the /// one this exists to fix. - async fn await_recorroboration(&self, retry: bool) { - if !retry { + /// + /// # How long to hold a refused session depends on WHO the round accused + /// + /// A [`RefusalReason::WriterContradicted`] round has already identified the writer as the + /// anomaly: four independently drawn peers agreed and it said something else at a height it did + /// not choose. There is nothing to wait for — a re-corroboration puts the same question to the + /// same writer — so the session ends at once and the ordinary reconnect path dials a different + /// peer. Nothing is discarded by ending it: a refused session subscribes nothing and wrote + /// nothing. + /// + /// Every other refusal waits [`RECORROBORATE_AFTER`], unchanged. A split says nothing about + /// this writer, and silence says nothing about anything — replacing a peer on either would + /// re-dial on a condition the peer may not be the cause of. + /// + /// This introduces no new constant and no new dial rate. The replacement is bounded by the + /// EXISTING ladder: an ended session shorter than [`HEALTHY_SESSION`] does not reset backoff, + /// so a permanent, locally-caused mismatch converges on one dial per [`BACKOFF_MAX`]. + async fn await_recorroboration(&self, refusal: Option) { + let Some(refusal) = refusal else { std::future::pending::<()>().await; + // `pending` never returns; this is unreachable and exists only to satisfy the type. + return; + }; + match refusal { + // Not `sleep(ZERO)`: a zero rung in the sleep record is a wait that was never taken, + // and the ladder is read positionally. + RefusalReason::WriterContradicted => {} + RefusalReason::Undecided | RefusalReason::WriterSilent => { + self.time.sleep(RECORROBORATE_AFTER).await; + } } - self.time.sleep(RECORROBORATE_AFTER).await; } /// Resolve once this session has held the replica STILL for [`STALL_AFTER`] while the chain diff --git a/crates/dig-wallet/src/sage/sync_supervisor/tests.rs b/crates/dig-wallet/src/sage/sync_supervisor/tests.rs index 7b599ec5..9f287ca7 100644 --- a/crates/dig-wallet/src/sage/sync_supervisor/tests.rs +++ b/crates/dig-wallet/src/sage/sync_supervisor/tests.rs @@ -90,6 +90,13 @@ struct Script { /// Heights the writer was asked about, in order, so a test can prove the writer never chose /// its own exam height. writer_asked_at: Mutex>, + /// When set, the writer's `header_hash_at` FAILS instead of answering. + /// + /// A separate knob from `writer_answer`, because a probe that errored and a peer that honestly + /// answered `None` reach `trust_for_session` by different routes, and a double that could only + /// express one of them could not show that both are treated as silence rather than as a lie + /// (dig_ecosystem#2868). + writer_errors: std::sync::atomic::AtomicBool, /// Sleep durations that NEVER return, so the timer they belong to cannot fire. /// /// This clock returns from every sleep immediately, which is what makes the backoff ladder @@ -231,6 +238,9 @@ impl SyncSession for ScriptedSession { async fn header_hash_at(&self, height: u32) -> Result, SyncError> { self.script.writer_asked_at.lock().unwrap().push(height); + if self.script.writer_errors.load(Ordering::SeqCst) { + return Err(SyncError::Peer("scripted header_hash_at failure".into())); + } Ok(*self.script.writer_answer.lock().unwrap()) } @@ -3140,6 +3150,329 @@ fn elevation_requires_both_a_verdict_and_the_writers_agreement() { assert!(!may_elevate(&split, None)); } +// --------------------------------------------------------------------------- +// Naming the refusal (dig_ecosystem#2868) — WHICH party the round accused +// --------------------------------------------------------------------------- + +/// Every verdict a round can reach, so the table below is exhaustive rather than representative. +fn every_verdict() -> Vec> { + vec![ + unanimous(HONEST_HASH), + Verdict::MajorityWithDissent { + answer: HONEST_HASH, + agreed: quorum::QUORUM_HOLD - 1, + dissenters: vec!["203.0.113.9".into()], + }, + Verdict::Split { + tallies: vec![2, 2], + }, + Verdict::Insufficient { + answered: 1, + required: quorum::CORROBORATION_FLOOR, + }, + ] +} + +/// **Proves:** naming the refusal did not WIDEN the write gate by one input. +/// +/// [`refusal`] is a richer function than [`may_elevate`] and sits beside it, so the risk it +/// introduces is not that it mislabels something — it is that the two drift and some input becomes +/// elevatable through the new door. This pins them together over the WHOLE input space: every +/// verdict crossed with agreeing / contradicting / silent. +/// +/// NEAREST WRONG IMPLEMENTATION: a `refusal` that returns `None` for a contradicting writer on a +/// unanimous round — i.e. option (a), the quorum's answer being taken as the session's — which the +/// supervisor would then read as "elevated" and hand the replica to a peer that just lied. +#[test] +fn refusal_agrees_with_may_elevate_on_every_input() { + for verdict in every_verdict() { + let round = CorroborationRound { + height: SETTLED_HEIGHT, + verdict, + }; + for answer in [Some(HONEST_HASH), Some(LIARS_HASH), None] { + assert_eq!( + refusal(&round, answer).is_none(), + may_elevate(&round, answer), + "the refusal classifier and the write gate disagreed about {round:?} / {answer:?}" + ); + } + } +} + +/// **Proves:** a round the QUORUM could not decide never accuses the writer, whatever the writer +/// said. +/// +/// FIXTURE DESIGN — the writer answers something DIFFERENT from every peer's answer. That is the +/// input that separates arm order from luck: a classifier comparing answers before checking the +/// verdict returns `WriterContradicted` here, and a fixture in which the writer happened to agree +/// would pass against it. +/// +/// NEAREST WRONG IMPLEMENTATION: matching on `(corroborated, writer_answer)` and testing the +/// answers first. It costs a re-dial on every split — the disagreement is with an undecided +/// quorum, not with the peer — which is the dial loop this ticket's third constraint forbids. +#[test] +fn an_undecided_round_never_accuses_the_writer() { + for verdict in [ + Verdict::Split { + tallies: vec![2, 2], + }, + Verdict::Insufficient { + answered: 1, + required: quorum::CORROBORATION_FLOOR, + }, + ] { + let round = CorroborationRound { + height: SETTLED_HEIGHT, + verdict, + }; + for answer in [Some(HONEST_HASH), Some(LIARS_HASH), None] { + assert_eq!( + refusal(&round, answer), + Some(RefusalReason::Undecided), + "an undecided round blamed the writer for answering {answer:?}" + ); + } + } + // The control: with a decisive quorum, the SAME contradicting answer is an accusation. Without + // this, the assertions above would also pass against a classifier that never accuses anyone. + let decisive = CorroborationRound { + height: SETTLED_HEIGHT, + verdict: unanimous(HONEST_HASH), + }; + assert_eq!( + refusal(&decisive, Some(LIARS_HASH)), + Some(RefusalReason::WriterContradicted) + ); +} + +/// **Proves:** silence is not a contradiction, and it is not counted as one. +/// +/// Both routes to silence are covered — an honest `Ok(None)` and a probe that errored — because +/// production folds the error into `None` at the call site, and a test exercising only the honest +/// route would not notice the fold being removed. +/// +/// NEAREST WRONG IMPLEMENTATION: `writer_answer != Some(agreed)` as the contradiction test, which +/// is true for `None` and so reports every slow peer as a liar. Its cost is not cosmetic: a liar is +/// replaced at once AND counted toward [`quorum::PERSISTENT_DISAGREEMENT_ROUNDS`], so a merely busy +/// peer would walk the node into a partition warning it has no evidence for. +#[test] +fn a_writer_that_could_not_answer_is_not_a_liar() { + let decisive = CorroborationRound { + height: SETTLED_HEIGHT, + verdict: unanimous(HONEST_HASH), + }; + assert_eq!( + refusal(&decisive, None), + Some(RefusalReason::WriterSilent), + "a writer that did not answer was treated as having disagreed" + ); + assert!( + !RefusalReason::WriterSilent.counts_as_disagreement(), + "silence was spent as evidence of peers disagreeing about settled chain state" + ); + // The control: the two refusals that ARE evidence still count, so the assertion above cannot + // be satisfied by a predicate that counts nothing and thereby disarms the partition warning. + assert!(RefusalReason::Undecided.counts_as_disagreement()); + assert!(RefusalReason::WriterContradicted.counts_as_disagreement()); +} + +/// Run one discovered-peer session against a scripted round, keeping the harness so the test can +/// read the supervisor's timers rather than only the replica. +/// +/// `writer_errors` makes the writer's probe FAIL rather than answer, which is the second route to +/// [`RefusalReason::WriterSilent`]. +async fn refused_sessions( + verdict: Verdict, + writer_answer: Option, + writer_errors: bool, +) -> (WalletDb, Harness) { + let db = WalletDb::open_in_memory().await.unwrap(); + let script = Script::new(); + *script.writer_answer.lock().unwrap() = writer_answer; + script.writer_errors.store(writer_errors, Ordering::SeqCst); + let hashes: Arc = + Arc::new(FixedHashes::unlocked(vec![Bytes32::new([7; 32])])); + + let harness = Harness::start_full( + db.clone(), + hashes, + script, + vec!["203.0.113.1:8444".into()], + PeerTrust::Discovered, + Some(ScriptedCorroborator::new(verdict)), + ) + .await; + // Two connects means the first session ENDED and the reconnect path ran — the state this + // ticket is about — rather than the supervisor merely having started. + harness + .until("a replacement session", |s| { + s.connects.load(Ordering::SeqCst) >= 2 + }) + .await; + (db, harness) +} + +/// **Proves:** a writer caught contradicting a decisive quorum is replaced WITHOUT the +/// [`RECORROBORATE_AFTER`] hold, and the quorum's answer still does not become a write. +/// +/// FIXTURE DESIGN — exactly ONE actor varies from the healthy case: the quorum is honest and +/// unanimous, and only the writer disagrees. An all-hostile fixture cannot see this property at +/// all, because there would be no decisive quorum for the writer to be the odd one out of. +/// +/// THE SECOND ASSERTION IS THE POINT, and it is about a PLACEMENT rather than an outcome. "The +/// session ended" is satisfied identically by the pre-existing 45-second path, so a test asserting +/// only that would pin a coincidence and stay green if the change were reverted. The observable +/// that separates them is which duration the supervisor asked its clock for. +/// +/// NEAREST WRONG IMPLEMENTATIONS: (a) taking the quorum's answer as the session's — caught by the +/// replica staying empty and `catch_up` never being called; (b) leaving the hold at +/// [`RECORROBORATE_AFTER`] for every refusal — caught by the sleep record. +#[tokio::test] +async fn a_contradicted_writer_ends_the_session_at_once_and_writes_nothing() { + let (db, harness) = refused_sessions(unanimous(HONEST_HASH), Some(LIARS_HASH), false).await; + + assert!( + !harness + .script + .slept + .lock() + .unwrap() + .contains(&RECORROBORATE_AFTER), + "a writer already caught contradicting the quorum was still held for the \ + re-corroboration interval" + ); + // Nothing was written, and nothing was even attempted: a refused session subscribes nothing, + // so the catch-up is never entered. Option (a) would fail both. + assert_eq!(harness.script.catch_up_count(), 0); + harness.stop().await; + assert_eq!(db.sync_state().await.unwrap().peak_height, None); +} + +/// **Proves:** the UNDECIDED round's behaviour is unchanged — it still waits +/// [`RECORROBORATE_AFTER`] before drawing a fresh sample. +/// +/// The control for the test above. Without it, "a contradicted session does not sleep" would also +/// pass against a supervisor that stopped waiting for ANY refusal, which would turn every split +/// into an immediate re-dial. +#[tokio::test] +async fn a_non_decisive_quorum_still_changes_nothing() { + let (db, harness) = refused_sessions( + Verdict::Split { + tallies: vec![2, 2], + }, + Some(HONEST_HASH), + false, + ) + .await; + + assert!( + harness + .script + .slept + .lock() + .unwrap() + .contains(&RECORROBORATE_AFTER), + "a split round stopped holding the session for the re-corroboration interval" + ); + assert_eq!(harness.script.catch_up_count(), 0); + harness.stop().await; + assert_eq!(db.sync_state().await.unwrap().peak_height, None); +} + +/// **Proves:** a writer whose probe FAILED is held for the unchanged interval, not replaced at +/// once — the supervisor-level half of `a_writer_that_could_not_answer_is_not_a_liar`. +/// +/// FIXTURE DESIGN — the quorum is DECISIVE, so the round has every ingredient of an accusation +/// except an answer to accuse. A split fixture would reach the same wait through +/// [`RefusalReason::Undecided`] and prove nothing about how silence is treated. +#[tokio::test] +async fn a_writer_whose_probe_failed_is_not_replaced_at_once() { + let (_db, harness) = refused_sessions(unanimous(HONEST_HASH), Some(LIARS_HASH), true).await; + + assert!( + harness + .script + .slept + .lock() + .unwrap() + .contains(&RECORROBORATE_AFTER), + "a writer that could not answer was replaced as though it had lied" + ); + harness.stop().await; +} + +/// **Proves:** a STANDING contradiction still reaches the partition warning — replacing the writer +/// promptly did not disarm the escalation that says a light client is looking at a fork or a +/// hostile peer set. +/// +/// The warning is asserted from the LOG, because that is the only place it exists: `splits` is the +/// supervisor's own local, and a test asserting on a counter it could reach would be asserting +/// against the production predicate rather than against the operator-visible outcome. +/// +/// NEAREST WRONG IMPLEMENTATION: not counting a contradiction toward the escalation — the tempting +/// simplification, since the writer is being replaced anyway. It would mean a node whose every +/// dialled peer contradicts the quorum reports nothing at all, which is precisely the silent +/// never-finishes-syncing failure the escalation exists to break. +#[tokio::test] +async fn persistent_contradiction_still_reaches_the_partition_warning() { + let capture = Capture::default(); + let guard = capture.install(); + + let (_db, harness) = refused_sessions(unanimous(HONEST_HASH), Some(LIARS_HASH), false).await; + let rounds = usize::try_from(quorum::PERSISTENT_DISAGREEMENT_ROUNDS).unwrap() + 1; + harness + .until("enough contradicted rounds to escalate", move |s| { + s.connects.load(Ordering::SeqCst) > rounds + }) + .await; + harness.stop().await; + + capture.assert_saw_the_supervisor(); + let log = capture.contents(); + assert!( + log.contains("peers persistently disagree about settled chain state"), + "a standing contradiction never escalated to the partition warning: {log}" + ); + // The specific line is emitted too, so an operator can tell WHICH party the node suspects — + // the diagnostic the two-way fold could not give them. + assert!( + log.contains("the writer contradicted an independent quorum"), + "the log never named the writer as the anomaly: {log}" + ); + drop(guard); +} + +/// **Proves:** a permanently contradicting peer set converges on the EXISTING backoff ladder, and +/// no new constant was introduced to bound the replacement. +/// +/// This is the ticket's third constraint. Replacing a writer immediately is a dial loop unless +/// something bounds it, and the something must be the ladder that is already there: a refused +/// session is far shorter than [`HEALTHY_SESSION`], so backoff is not reset and doubles toward +/// [`BACKOFF_MAX`]. +/// +/// NEAREST WRONG IMPLEMENTATION: resetting the ladder on a refused session, or ending it through a +/// path that counts as healthy — either produces a sustained one-dial-per-second rate against the +/// introducers for as long as the condition lasts. +#[tokio::test] +async fn repeated_contradiction_climbs_the_existing_backoff() { + let (_db, harness) = refused_sessions(unanimous(HONEST_HASH), Some(LIARS_HASH), false).await; + harness + .until("four backoff rungs", |s| s.slept.lock().unwrap().len() >= 4) + .await; + let delays = harness.script.slept.lock().unwrap().clone(); + harness.stop().await; + + for (i, base) in [1u64, 2, 4, 8].iter().enumerate() { + let d = delays[i]; + assert!( + d >= Duration::from_millis(base * 800) && d <= Duration::from_millis(base * 1200), + "rung {i} should be the EXISTING ladder's ~{base}s (+/-20%), got {d:?} — a new \ + constant, or a reset ladder, would show up here" + ); + } +} + // --------------------------------------------------------------------------- // The union source (dig_ecosystem#2823) — custody ∪ externally-registered keys // --------------------------------------------------------------------------- From 7be0ca2419168c9ce67f766fcc86fbe04a7f2207 Mon Sep 17 00:00:00 2001 From: Michael Taylor Date: Thu, 13 Aug 2026 15:34:09 -0700 Subject: [PATCH 3/7] fix(wallet): report a behind replica's figure as stale rather than current MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `Source::Db` arms of `balance_for_address` and `coins_for_address` set `synced` to the literal `true`, so every replica-served answer claimed to be at the tip. That claim was never checked against anything: the flag that selects the tier is `initial_sync_complete`, which LATCHES — it records that a catch-up once finished, and only a backwards chain move clears it — so a replica hundreds of blocks behind still routes to the replica and still reported its figure as current. `synced` is now measured, from the same `is_following` predicate `control.wallet.syncStatus` reports its phase from, so the two endpoints cannot disagree about the same moment. A behind replica keeps SERVING, and keeps reporting its real `peak_height`: `synced: false` beside a height means "this figure is real, as of that height", which is a usable answer, not a withheld one. Fallback answers are unchanged — `false` / `null`. The `db_synced` axis of `routing::route` is deliberately NOT removed. Measured on the live node: `peak_height = 9140640`, `initial_sync_complete = 0`, and zero coin rows. `peak_height` advances from `new_peak_wallet` independently of any coin being applied, so a present peak is evidence about the chain and never about this replica's coverage. Serving that state would render as "Balance: 0, correct as of block 9,140,640" for a wallet holding 1.599 XCH. `an_incomplete_catch_up_never_serves_a_dated_zero_from_the_replica` holds the axis in place with a fixture built in exactly that state. Version coupling: an un-upgraded dig-app refuses any `!synced` reading, so it will stop showing balances it shows today. The dig-app half of #2869 lands first. Refs dig_ecosystem#2869 Co-Authored-By: Claude --- crates/dig-wallet/src/sage/rpc.rs | 233 ++++++++++++++++++++++++++++-- 1 file changed, 220 insertions(+), 13 deletions(-) diff --git a/crates/dig-wallet/src/sage/rpc.rs b/crates/dig-wallet/src/sage/rpc.rs index 9f848e5e..dbad51eb 100644 --- a/crates/dig-wallet/src/sage/rpc.rs +++ b/crates/dig-wallet/src/sage/rpc.rs @@ -189,7 +189,9 @@ pub struct WalletCoinsResult { pub coins: Vec, /// Which tier produced these coins. pub source: Source, - /// Whether THIS answer reflects a fully-synced local view (only a [`Source::Db`] answer can be). + /// Whether THIS answer is CURRENT — see [`WalletBalanceResult::synced`], of which this is the + /// unreduced twin and which carries the full contract. `false` beside a [`Self::peak_height`] + /// is a real coin set as of that height, not an unknown one. pub synced: bool, /// The chain peak height THIS answer reflects, when known. pub peak_height: Option, @@ -316,12 +318,19 @@ pub struct WalletBalanceResult { /// /// Additive per §5.1: a consumer that ignores it parses unchanged. pub source: Source, - /// Whether THIS answer reflects a fully-synced local view. - /// - /// Derived from the tier, never from the DB flag independently: only a - /// [`Source::Db`] answer can be synced, because only that tier read the local - /// replica. A fallback answer reports `false` however caught-up the DB happens to - /// be — the DB's state does not describe an answer the DB did not give (#2233). + /// Whether THIS answer is CURRENT — the replica produced it AND the replica is following the + /// chain right now. + /// + /// Both clauses are required. Only a [`Source::Db`] answer can be current, because only that + /// tier read the local replica: a fallback answer reports `false` however caught-up the DB + /// happens to be, since the DB's state does not describe an answer the DB did not give + /// (#2233). + /// + /// And a [`Source::Db`] answer is not current merely by being served. The flag that chose the + /// tier, `initial_sync_complete`, LATCHES — it records that a catch-up once finished — so a + /// replica hundreds of blocks behind still routes here. `false` alongside a + /// [`Self::peak_height`] therefore means *this figure is real, and it is as of that height*, + /// which is a usable answer; it does not mean the figure is unknown (dig_ecosystem#2869). pub synced: bool, /// The chain peak height THIS answer reflects, when known — `None` for a /// [`Source::Fallback`] answer, whose figure came from the oracle's chain view, not @@ -805,6 +814,35 @@ impl WalletBackend { self } + /// The node's OWN Chia peer tier — peers held, and the peak they announced. + /// + /// A local read of the transport's cached state: it makes no oracle call, so it opens none of + /// the egress this file refuses elsewhere, and it is cheap enough to take on an ordinary read + /// path. It is the second opinion every freshness claim needs, because the replica's own peak + /// cannot say how far behind the replica is. + async fn chain_peer_tier(&self) -> super::fallback::ChainPeerTier { + match self.chain_peer_tier_override { + Some(fixed) => fixed, + None => self.fallback.peer_tier().await, + } + } + + /// Whether a figure taken from the replica AT `peak_height` may be reported as CURRENT. + /// + /// `db_synced` — which chose the replica in the first place — cannot answer this. It is + /// `initial_sync_complete`, which records that a catch-up once FINISHED and is cleared only by + /// a backwards chain move; a replica hundreds of blocks behind still satisfies it. Reporting + /// `synced: true` on that basis told a client a stale balance was settled, which is the + /// money-adjacent falsehood dig_ecosystem#2869 exists to remove. + /// + /// It reuses [`super::sync_supervisor::is_following`] — the SAME predicate + /// `control.wallet.syncStatus` reports its phase from — so a client cannot be told `synced` by + /// one endpoint and `syncing` by the other about the same moment. An unobservable peer tier is + /// never an accusation there and is not one here. + async fn replica_answer_is_current(&self, peak_height: Option) -> bool { + super::sync_supervisor::is_following(peak_height, self.chain_peer_tier().await.peak_height) + } + /// The chain-sync supervisor's handle, if one is running. pub fn sync_handle(&self) -> Option<&super::sync_supervisor::SyncHandle> { self.sync_handle.as_ref() @@ -838,10 +876,7 @@ impl WalletBackend { // This is a local read of the transport's own state plus a cached peak the peers pushed; // it makes no oracle call, so it does not open the egress path this method's doc above // refuses for the peak. - let tier = match self.chain_peer_tier_override { - Some(fixed) => fixed, - None => self.fallback.peer_tier().await, - }; + let tier = self.chain_peer_tier().await; match &self.sync_handle { Some(h) => h.status(&self.db, tier).await, None => super::sync_supervisor::status_without_supervisor(&self.db, tier).await, @@ -1085,7 +1120,10 @@ impl WalletBackend { balance, pending, source, - synced: true, + // Measured, not assumed: this arm reports what the replica HOLDS, and whether + // that is current is a different question from whether the replica was + // eligible to answer (dig_ecosystem#2869). + synced: self.replica_answer_is_current(peak_height).await, peak_height, }) } @@ -1215,7 +1253,10 @@ impl WalletBackend { Ok(WalletCoinsResult { coins, source, - synced: true, + // The same measurement as the balance read's: this is that answer unreduced, + // and the caller building a spend on it is the one who can least afford to be + // told a stale coin set is current. + synced: self.replica_answer_is_current(peak_height).await, peak_height, }) } @@ -5848,6 +5889,172 @@ mod tests { ); } + // ----------------------------------------------------------------------- + // `synced` describes THIS answer's freshness (dig_ecosystem#2869) + // ----------------------------------------------------------------------- + + /// The replica's peak in these fixtures, and a peer tier far enough ahead to be unambiguous. + /// + /// The gap is drawn FROM the production bound rather than invented: `FOLLOWING_TOLERANCE` is + /// four blocks, and a fixture sitting just inside it would assert the tolerance instead of the + /// behaviour. 530 is the distance measured on the live node this ticket came from. + const REPLICA_PEAK: u32 = 9_140_640; + const PEERS_AHEAD_BY: u32 = 530; + + /// A tier whose peers announced a peak `PEERS_AHEAD_BY` blocks past the replica's. + fn peers_ahead_of_the_replica() -> super::super::fallback::ChainPeerTier { + super::super::fallback::ChainPeerTier { + peer_count: Some(5), + peak_height: Some(REPLICA_PEAK + PEERS_AHEAD_BY), + } + } + + /// **Proves:** a replica that completed a catch-up and then fell BEHIND reports its figure as + /// stale — `synced: false` — while still disclosing the height that figure is as of. + /// + /// THE DEFECT THIS PINS. `synced` was the literal `true` in the `Source::Db` arm, so it said + /// "current" about every replica-served answer for as long as the process lived. It is not a + /// small overstatement: `db_synced` is `initial_sync_complete`, which latches once and is + /// cleared only by a backwards chain move, so a replica 530 blocks behind still routes here and + /// still claimed to be at the tip. + /// + /// FIXTURE DESIGN — the answer must stay SERVED. A wrong fix suppresses the whole reading, or + /// blanks the peak, and a test asserting only `!synced` would pass against it. Asserting the + /// balance AND the peak alongside is what pins "stale but honest and still useful" rather than + /// "withheld". + #[tokio::test] + async fn a_behind_replica_serves_its_figure_and_says_it_is_not_current() { + let db = db_with_owned_derivation(true, Some(REPLICA_PEAK)).await; + db.upsert_coin(&coin_at_ph("aa", &owned_ph(), 1_599_179_999_973, Some(1), None)) + .await + .unwrap(); + let be = WalletBackend::new(db, Arc::new(EmptyFallback), WalletConfig::default()) + .with_chain_peer_tier_for_tests(peers_ahead_of_the_replica()); + + let result = be + .balance_for_address(&owned_address(), BalanceAsset::Xch) + .await + .unwrap(); + assert_eq!(result.source, Source::Db, "the replica stopped serving"); + assert_eq!(result.balance, 1_599_179_999_973, "the figure was withheld"); + assert_eq!( + result.peak_height, + Some(REPLICA_PEAK), + "a stale answer must still say WHAT it is as of" + ); + assert!( + !result.synced, + "a replica {PEERS_AHEAD_BY} blocks behind reported its figure as current" + ); + } + + /// **Proves:** the honesty fix did not make every replica answer stale — a replica level with + /// its peers still reports `synced: true`. + /// + /// The control. Without it, the test above is satisfied by `synced: false` hardcoded in place + /// of `synced: true`, which trades one literal for another and would make an upgraded client + /// distrust every reading the node ever gives it. + #[tokio::test] + async fn a_replica_level_with_its_peers_still_reports_synced() { + let db = db_with_owned_derivation(true, Some(REPLICA_PEAK)).await; + db.upsert_coin(&coin_at_ph("aa", &owned_ph(), 42, Some(1), None)) + .await + .unwrap(); + let be = WalletBackend::new(db, Arc::new(EmptyFallback), WalletConfig::default()) + .with_chain_peer_tier_for_tests(super::super::fallback::ChainPeerTier { + peer_count: Some(5), + peak_height: Some(REPLICA_PEAK), + }); + + let result = be + .balance_for_address(&owned_address(), BalanceAsset::Xch) + .await + .unwrap(); + assert!(result.synced, "a replica at the tip was reported as stale"); + assert_eq!(result.peak_height, Some(REPLICA_PEAK)); + } + + /// **Proves:** the coin read makes the SAME claim as the balance read about the same replica. + /// + /// They are the same answer reduced differently, and a caller building a spend reads this one. + /// A fix applied to only the balance arm would leave the spend path being told a 530-block-old + /// coin set was current — the more expensive half of the two to be wrong about. + #[tokio::test] + async fn the_coin_read_reports_the_same_freshness_as_the_balance_read() { + let db = db_with_owned_derivation(true, Some(REPLICA_PEAK)).await; + db.upsert_coin(&coin_at_ph("aa", &owned_ph(), 42, Some(1), None)) + .await + .unwrap(); + let be = WalletBackend::new(db, Arc::new(EmptyFallback), WalletConfig::default()) + .with_chain_peer_tier_for_tests(peers_ahead_of_the_replica()); + + let coins = be + .coins_for_address(&owned_address(), BalanceAsset::Xch) + .await + .unwrap(); + assert_eq!(coins.source, Source::Db); + assert_eq!(coins.coins.len(), 1, "the coin set was withheld"); + assert_eq!(coins.peak_height, Some(REPLICA_PEAK)); + assert!(!coins.synced, "the spend path was told a stale coin set was current"); + } + + /// **Proves:** a fallback answer is unchanged — it never borrows the replica's freshness or + /// its height, however caught-up the replica happens to be. + #[tokio::test] + async fn a_fallback_answer_still_claims_neither_freshness_nor_a_height() { + let db = db_with_owned_derivation(false, Some(REPLICA_PEAK)).await; + // A LIVE fallback: an unreachable one errors before it can construct an answer, and this + // test is about the answer's fields. + let be = WalletBackend::new(db, Arc::new(MockFallback::default()), WalletConfig::default()) + .with_chain_peer_tier_for_tests(peers_ahead_of_the_replica()); + + let result = be + .balance_for_address(&owned_address(), BalanceAsset::Xch) + .await + .unwrap(); + assert_eq!(result.source, Source::Fallback); + assert!(!result.synced); + assert_eq!(result.peak_height, None); + } + + /// **Proves (dig_ecosystem#2869):** an incomplete catch-up NEVER serves from the replica, even + /// though the replica has a peak height. + /// + /// READ THIS BEFORE REMOVING THE `db_synced` AXIS FROM [`routing::route`]. #2869's Scope section + /// asks for exactly that, on the premise that a behind replica holding the user's coins is not + /// consulted. The premise does not hold: `db_synced` is `initial_sync_complete`, which latches, + /// so a behind-but-once-synced replica already routes to [`Source::Db`]. What the axis actually + /// excludes is a replica that has never finished a catch-up — and that replica holds NOTHING. + /// + /// FIXTURE DESIGN — the peak is deliberately PRESENT and the coin table deliberately EMPTY, + /// which is the exact state measured on the live node (`peak_height = 9140640`, + /// `initial_sync_complete = 0`, zero coin rows). `new_peak_wallet` advances the peak + /// independently of any coin being applied, so a present peak is evidence about the CHAIN and + /// never about this replica's coverage. Serving that state renders to the user as *"Balance: 0, + /// correct as of block 9,140,640"* for a wallet holding 1.599 XCH — well-formed, precisely + /// dated, and false. A fixture leaving the peak unset would pass against that implementation + /// and prove nothing. + #[tokio::test] + async fn an_incomplete_catch_up_never_serves_a_dated_zero_from_the_replica() { + let db = db_with_owned_derivation(false, Some(REPLICA_PEAK)).await; + assert!( + !db.is_synced().await.unwrap(), + "the fixture must be the never-caught-up replica" + ); + assert_eq!( + db.sync_state().await.unwrap().peak_height, + Some(REPLICA_PEAK), + "the fixture must carry a peak; without one it proves nothing" + ); + + assert_eq!( + routing::route(db.is_synced().await.unwrap(), true), + Source::Fallback, + "a replica with a peak but no completed catch-up was served as authoritative; it holds \ + no coins, so that answer is a dated zero for a funded wallet" + ); + } + /// The peak comes from the node's own replica when it has one. #[tokio::test] async fn the_peak_is_the_replicas_when_the_replica_has_one() { From da9dd88efb7b70d408565dbfdb8e9fb53c1fd935 Mon Sep 17 00:00:00 2001 From: Michael Taylor Date: Thu, 13 Aug 2026 15:54:33 -0700 Subject: [PATCH 4/7] docs(spec): state the three refusals and what `synced` means per tier MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SPEC described the refusal as one outcome ("no corroborated answer") and `synced` as a property a `"db"` answer simply has. Both are now false of the implementation, and a spec left describing the two-way fold is the version a reimplementation would build. Adds the three-refusal table (undecided / writer contradicted / writer silent), the load-bearing arm order, the prohibition on adopting the quorum's answer as a write, and the requirement that the replacement be bounded by the existing reconnect ladder rather than a new constant. §18.7b now states that `synced` is MEASURED, that a `"db"` answer with `synced: false` beside a `peak_height` is a real figure that MUST still be served, and why the latching `initial_sync_complete` cannot answer the question. The acceptance script's gate 3 stopped requiring the replica to be within 50 blocks of the tip: a behind replica must still serve, so failing the run on the distance would fail the correct behaviour. It now requires the replica to have a height to answer as of, reports the distance as context, and asserts the falsehood that was actually removed — a far-behind replica may serve, but may never report `synced: true`. Refs dig_ecosystem#2868, dig_ecosystem#2869 Co-Authored-By: Claude --- Cargo.lock | 2 +- Cargo.toml | 2 +- SPEC.md | 53 ++++++++++++++++--- crates/dig-wallet/src/sage/rpc.rs | 25 ++++++--- crates/dig-wallet/src/sage/sync_supervisor.rs | 11 ++-- scripts/acceptance-wallet-balance.sh | 30 ++++++++--- 6 files changed, 95 insertions(+), 28 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index c90f35b0..44c28791 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2307,7 +2307,7 @@ dependencies = [ [[package]] name = "dig-node-service" -version = "0.117.1" +version = "0.118.0" dependencies = [ "async-trait", "axum", diff --git a/Cargo.toml b/Cargo.toml index aee2df4e..5c2384d8 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -32,7 +32,7 @@ edition = "2021" # the ROOT manifest (`[workspace.package].version`), so it MUST be set here for a # release to fire (§3.6). The library crates (dig-node-core/dig-runtime/dig-wallet) # keep their own independent versions — only the released binary tracks the workspace version. -version = "0.117.1" +version = "0.118.0" # Release hardening, matching digstore: keep integer-overflow checks ON in release. # The node parses untrusted serialized input and does offset/length arithmetic over diff --git a/SPEC.md b/SPEC.md index b8845516..04b436b3 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1505,7 +1505,7 @@ lowercase 64-hex; a capsule reference is `storeId:rootHash`. Malformed refs yiel | `control.hostedStores.status` | `store` = `storeId[:rootHash]` | `store_id`, `pinned`, `capsule_count`, `total_bytes`, `capsules[]` | | `control.sync.status` | — | `available` (always `true` — the chunked capsule download needs no identity), `method: "chunked-capsule-download-with-section-21-clone-fallback"`, `identity_loaded`, `pinned_total`, `pinned_synced`, `whole_store_trigger_supported` (`true` — a store id alone is enough) | | `control.sync.trigger` | `store` = `storeId[:rootHash]`, or `store_id` [+ `root`] — the root is OPTIONAL; without one the node resolves the store's CHAIN-ANCHORED tip and syncs that generation | `status: "synced"`, `root`, `size_bytes`, `served_root` | -| `control.wallet.balance` | `address` (bech32m string), `asset` (`"xch"` \| `"dig"`, default `"xch"`) | `balance` (confirmed, spendable — JSON NUMBER, u64 base units), `pending` (unspent + unconfirmed — JSON NUMBER, u64 base units), `source` (`"db"` \| `"fallback"` — which tier produced the figure, §18.7b), `synced` (bool), `peak_height` (`u32` or `null`). Matches `dig-node-control-interface` 0.3.0's `WalletBalanceResult { balance: u64, pending: u64, .. }` and dig-app's `BalanceResponse { balance: u64 }` — a Rust-to-Rust numeric contract, never a decimal string. The wallet backend tracks the base-unit total as `u128` (headroom for summed intermediate math); the wire boundary saturating-casts to `u64` (a single address's balance can never exceed `u64::MAX` mojos, ~18.4M XCH). READ-ONLY chain read of a PUBLIC address (no seed/signing key). Reuses the B.6 sync-state routing: the local DB when the address is the wallet's own and the DB is synced, else the coinset fallback. Per §18.7b, `source`/`synced`/`peak_height` describe the TIER that answered: a `"db"` answer reports `synced: true` and the node's own peak; a `"fallback"` answer reports `synced: false` and `peak_height: null`. This is an OPEN read (`is_open_control_read`, no token); the cheap local-DB fast path is unbounded, but the EXPENSIVE coinset-fallback leg is subject to a GLOBAL token-bucket rate bound (defense-in-depth against an open-read amplification/oracle sweep — #1957): a burst of arbitrary-address fallback reads beyond the bound is refused with `WALLET_RATE_LIMITED` (§10), while any single honest read (DB fast path or one fallback) always succeeds. `$DIG` scopes by the canonical CAT asset id `digstore_chain::dig::DIG_ASSET_ID`. A synced empty address is a SUCCESS `{balance:0, synced:true}`, never an error; the read-failure shapes are DISTINCT errors `WALLET_NO_CHAIN_SOURCE`/`WALLET_NOT_SYNCED`/`WALLET_READ_FAILED`/`WALLET_RATE_LIMITED` (§10), never a fabricated `0`. `INVALID_PARAMS` on a missing/malformed `address` or a bad `asset`. | +| `control.wallet.balance` | `address` (bech32m string), `asset` (`"xch"` \| `"dig"`, default `"xch"`) | `balance` (confirmed, spendable — JSON NUMBER, u64 base units), `pending` (unspent + unconfirmed — JSON NUMBER, u64 base units), `source` (`"db"` \| `"fallback"` — which tier produced the figure, §18.7b), `synced` (bool), `peak_height` (`u32` or `null`). Matches `dig-node-control-interface` 0.3.0's `WalletBalanceResult { balance: u64, pending: u64, .. }` and dig-app's `BalanceResponse { balance: u64 }` — a Rust-to-Rust numeric contract, never a decimal string. The wallet backend tracks the base-unit total as `u128` (headroom for summed intermediate math); the wire boundary saturating-casts to `u64` (a single address's balance can never exceed `u64::MAX` mojos, ~18.4M XCH). READ-ONLY chain read of a PUBLIC address (no seed/signing key). Reuses the B.6 sync-state routing: the local DB when the address is the wallet's own and the DB is synced, else the coinset fallback. Per §18.7b, `source`/`synced`/`peak_height` describe the TIER that answered: a `"db"` answer reports the node's own peak and reports `synced: true` only while the replica is FOLLOWING the chain, so a behind-but-once-synced replica answers `synced: false` WITH its real `peak_height` rather than presenting a stale figure as current; a `"fallback"` answer reports `synced: false` and `peak_height: null`. This is an OPEN read (`is_open_control_read`, no token); the cheap local-DB fast path is unbounded, but the EXPENSIVE coinset-fallback leg is subject to a GLOBAL token-bucket rate bound (defense-in-depth against an open-read amplification/oracle sweep — #1957): a burst of arbitrary-address fallback reads beyond the bound is refused with `WALLET_RATE_LIMITED` (§10), while any single honest read (DB fast path or one fallback) always succeeds. `$DIG` scopes by the canonical CAT asset id `digstore_chain::dig::DIG_ASSET_ID`. A synced empty address is a SUCCESS `{balance:0, synced:true}`, never an error (and `synced` there means MEASURED-current, never merely eligible); the read-failure shapes are DISTINCT errors `WALLET_NO_CHAIN_SOURCE`/`WALLET_NOT_SYNCED`/`WALLET_READ_FAILED`/`WALLET_RATE_LIMITED` (§10), never a fabricated `0`. `INVALID_PARAMS` on a missing/malformed `address` or a bad `asset`. | | `control.wallet.coins` | `address` (bech32m string), `asset` (`"xch"` \| `"dig"`, default `"xch"`) | `coins` (array of `{coin_id, asset, amount, parent_coin_info, puzzle_hash, created_height, spent_height}`; all hashes lowercase 64-hex unprefixed, `amount` a JSON NUMBER in base units), `source`, `synced`, `peak_height` — the tier fields carrying exactly their `control.wallet.balance` meanings (§18.7b). The UNSPENT coins at the address for the asset, i.e. the read a caller building a spend needs; a balance is this read reduced to a sum, which is why the two take identical params. Coins seen only in the mempool are INCLUDED with `created_height: null`, so the caller decides what is spendable for its purpose rather than the node hiding one. `coins: []` MUST mean a chain WAS consulted and the address holds nothing; every way of failing to consult one is a DISTINCT error (`WALLET_NO_CHAIN_SOURCE`/`WALLET_NOT_SYNCED`/`WALLET_READ_FAILED`/`WALLET_RATE_LIMITED`, §10), NEVER an empty list — an empty list would tell a holder of funds that they hold none, and a spend built on it refuses with an untrue shortfall. OPEN read, same global fallback rate bound as the balance. `INVALID_PARAMS` on a missing/malformed `address` or a bad `asset`. | | `control.wallet.coinById` | `coin_id` (64 lowercase-hex, an optional `0x` prefix TOLERATED and normalized away) | `coin` (`{coin_id, asset, amount, parent_coin_info, puzzle_hash, created_height, spent_height}` or `null`), `source`, `synced`, `peak_height` — the tier fields carrying exactly their `control.wallet.balance` meanings (§18.7b); `source` is always `"fallback"`, `synced` always `false` and `peak_height` always `null`, because the answer never comes from the local replica (a miss there means "this node does not watch that coin", which is NOT absence). ONE coin by its own id, SPENT OR UNSPENT — the read a caller polling a spend needs and `control.wallet.coins` structurally cannot give: a created DID coin sits at nobody's wallet address, and a spent funding coin is gone from every unspent list. `asset` in the record is ALWAYS `null`: a coin id alone does not reveal whether a coin is XCH, a CAT or a singleton — that needs the puzzle, which this read never inspects — so naming one would assert a classification the node did not verify. A returned record MUST be bound to the id asked for: a coin id is self-certifying (`SHA256(parent ‖ puzzle_hash ‖ amount)`), so a source that answers with a DIFFERENT coin is a `WALLET_READ_FAILED` (§10) — never that coin's record, and never `coin: null`. `coin: null` MUST mean a chain source ANSWERED and reported no such coin; every way of failing to get an answer is a DISTINCT error (`WALLET_NO_CHAIN_SOURCE`/`WALLET_READ_FAILED`/`WALLET_RATE_LIMITED`, §10), NEVER a `null` — a `null` for an outage would tell a caller polling a mint that its coin does not exist, so a pending mint reads as awaiting forever. A caller MUST treat `null` as "not seen yet" and keep polling, not as "never happened". OPEN read (no token), same global fallback rate bound as the balance. `INVALID_PARAMS` on a missing/malformed `coin_id`, refused BEFORE any network call — an unanswerable question and a chain that answered "no" must never wear the same shape; the well-formedness rule is `dig-node-control-interface`'s own `WalletCoinByIdParams::validated()`, consumed rather than restated. | | `control.wallet.coinSpend` | `coin_id` (64 lowercase-hex, an optional `0x` prefix TOLERATED and normalized away) | `spend` (`{coin, puzzle_reveal, solution}` or `null`), `source`, `synced`, `peak_height` -- the tier fields carrying exactly their `control.wallet.coinById` meanings and the same always-`"fallback"` / always-`false` / always-`null` values, for the same reason. THE SPEND THAT SPENT ONE COIN, named by that coin's own id (a spend has no id of its own on chain). A coin record carries a puzzle HASH and says only that a coin is gone; the puzzle REVEAL and the solution exist only here, and they are what a caller reconstructing a lineage -- following a dig-profile's DID singleton forward -- needs. `coin` is the full record shape `control.wallet.coinById` returns, with `asset` ALWAYS `null` (this read classifies nothing) and `spent_height` ALWAYS non-null (a spend of a coin nothing calls spent is a contradiction; the node MUST fail closed rather than emit one). The node MUST verify that `puzzle_reveal` tree-hashes to `coin.puzzle_hash` and MUST refuse -- `WALLET_READ_FAILED` (§10) -- when it does not or will not parse: the reveal comes from an unauthenticated peer, a puzzle hash IS the reveal's CLVM tree hash, so the lie is locally detectable and a caller would otherwise curry a forged program into the spend it signs. The returned spend MUST be bound to the id asked for, by the same self-certifying coin-id recomputation `control.wallet.coinById` requires. `spend: null` MUST mean a chain source ANSWERED and holds no spend of that coin -- it is UNSPENT, or unknown; distinguishing those two is `control.wallet.coinById`'s job. Every way of failing to get an answer is a DISTINCT error, NEVER `null`: a caller walking a lineage reads "no spend" as *this is the tip* and stops, so a failure disguised as absence yields a spend built against a superseded singleton, and a mint poll reads it as "my funding coin is still there" and funds the same mint twice. OPEN read (no token), same global fallback rate bound. `INVALID_PARAMS` on a missing/malformed `coin_id`, refused BEFORE any network call; the rule is `dig-node-control-interface`'s own `WalletCoinSpendParams::validated()`, consumed rather than restated. | @@ -4042,6 +4042,35 @@ The retry obligation on a Split is discharged by ENDING the refused session: cor session, so a refused session that is never ended is a refusal that never expires and no fresh sample is ever drawn. An implementation MUST end a session that failed corroboration rather than holding it. +**A refusal MUST name WHICH party it accuses, and the three refusals are NOT interchangeable.** A round +that does not elevate its writer is one of exactly three things, and an implementation MUST distinguish +them: + +| Refusal | The round | Required behaviour | +|---|---|---| +| **Undecided** — Split or Insufficient, or corroboration unavailable | The truth is unknown, so nothing is known about the writer | Write NOTHING. Count toward `PERSISTENT_DISAGREEMENT_ROUNDS`. Hold the session for `RECORROBORATE_AFTER`, then end it. | +| **Writer contradicted** — a verdict was reached and the writer answered something ELSE at `H` | The WRITER is the anomaly | Write NOTHING. Count toward `PERSISTENT_DISAGREEMENT_ROUNDS`. END THE SESSION AT ONCE so a different peer is dialled. | +| **Writer silent** — a verdict was reached and the writer did not answer at all | Nothing is evidenced | Write NOTHING. MUST NOT count toward `PERSISTENT_DISAGREEMENT_ROUNDS`. Hold for `RECORROBORATE_AFTER`, then end. | + +The undecided round MUST be tested FIRST: without a verdict there is nothing for the writer to have +contradicted, and an implementation that compares answers first accuses an honest writer on every split. + +**A contradicted writer is replaced; the quorum's answer is NOT adopted.** When four independently drawn +peers agree and the writer disagrees, the evidence points at the writer, so keeping it for a further +`RECORROBORATE_AFTER` preserves precisely the peer that failed while the replica goes unwritten. The +session MUST therefore end immediately. It MUST NOT become a write: the quorum's answer is settled at the +deliberately lagged `H`, so recording it would put a chain fact into `sync_state` with no write authority +holder — the bypass this whole section exists to prevent — and would in any case never track the tip. + +**A probe that could not be answered is NOT evidence.** An implementation MUST NOT treat a +`header_hash_at` error, or an honest absence of an answer, as a contradiction. Silence is what a slow or +busy peer looks like, and counting it walks a node toward a partition warning it has no evidence for. + +**The replacement MUST be bounded by the EXISTING reconnect ladder, with no new constant.** A refused +session is far shorter than `HEALTHY_SESSION`, so the backoff is not reset and doubles toward +`BACKOFF_MAX`. A permanent mismatch — including one caused locally rather than by any peer — therefore +converges on one dial per `BACKOFF_MAX`, never a dial loop. + **Reads that MUST be verified rather than voted on.** Voting on a locally decidable fact wastes round trips and, worse, lets a majority overrule arithmetic. The following are SELF-VERIFYING and MUST be checked locally, never put to a quorum: @@ -4343,11 +4372,23 @@ answered, and MUST derive every freshness field from that tier. means the queried address WAS DISCLOSED off-node — a fact a caller on a metered or private connection has a legitimate interest in. The field is additive per §5.1: a consumer that does not read it parses unchanged. -- **`synced` and `peak_height` are properties of the tier.** A `"db"` answer reports `synced: true` - and the replica's own peak. A `"fallback"` answer reports `synced: false` and `peak_height: null`, - **regardless of the local DB's state** — the DB neither produced that figure nor bounds its - freshness, so its flag and peak say nothing about it. Implementations MUST NOT read those two - fields outside the tier decision. +- **`synced` and `peak_height` are properties of the tier, and `synced` is MEASURED.** A `"db"` + answer reports the replica's own peak, and reports `synced: true` **only if the replica is + FOLLOWING the chain right now** — the same `is_following` test `control.wallet.syncStatus` derives + its phase from, so the two endpoints MUST NOT disagree about the same moment. An implementation + MUST NOT report `synced: true` merely because the replica was eligible to answer: the flag that + selects the tier, `initial_sync_complete`, LATCHES — it records that a catch-up once finished and + is cleared only by a backwards chain move — so a replica hundreds of blocks behind still routes to + `"db"`, and reporting its figure as current tells a client a stale balance is settled. + + A `"db"` answer with `synced: false` alongside a `peak_height` is a REAL figure, as of that height, + and MUST still be served. It does not mean the figure is unknown, and an implementation MUST NOT + withhold it or blank the peak: `peak_height` is what makes a stale answer usable rather than + merely suspect. + + A `"fallback"` answer reports `synced: false` and `peak_height: null`, **regardless of the local + DB's state** — the DB neither produced that figure nor bounds its freshness, so its flag and peak + say nothing about it. Implementations MUST NOT read those two fields outside the tier decision. - **Rationale — this is the falsifiability instrument for §18.6.** A success criterion phrased as a flag value rather than as the path taken is satisfiable with the goal unmet: once the §18.6 sync loop sets `initial_sync_complete`, a read still served by the oracle would report itself as a diff --git a/crates/dig-wallet/src/sage/rpc.rs b/crates/dig-wallet/src/sage/rpc.rs index dbad51eb..f553f943 100644 --- a/crates/dig-wallet/src/sage/rpc.rs +++ b/crates/dig-wallet/src/sage/rpc.rs @@ -5925,9 +5925,15 @@ mod tests { #[tokio::test] async fn a_behind_replica_serves_its_figure_and_says_it_is_not_current() { let db = db_with_owned_derivation(true, Some(REPLICA_PEAK)).await; - db.upsert_coin(&coin_at_ph("aa", &owned_ph(), 1_599_179_999_973, Some(1), None)) - .await - .unwrap(); + db.upsert_coin(&coin_at_ph( + "aa", + &owned_ph(), + 1_599_179_999_973, + Some(1), + None, + )) + .await + .unwrap(); let be = WalletBackend::new(db, Arc::new(EmptyFallback), WalletConfig::default()) .with_chain_peer_tier_for_tests(peers_ahead_of_the_replica()); @@ -5995,7 +6001,10 @@ mod tests { assert_eq!(coins.source, Source::Db); assert_eq!(coins.coins.len(), 1, "the coin set was withheld"); assert_eq!(coins.peak_height, Some(REPLICA_PEAK)); - assert!(!coins.synced, "the spend path was told a stale coin set was current"); + assert!( + !coins.synced, + "the spend path was told a stale coin set was current" + ); } /// **Proves:** a fallback answer is unchanged — it never borrows the replica's freshness or @@ -6005,8 +6014,12 @@ mod tests { let db = db_with_owned_derivation(false, Some(REPLICA_PEAK)).await; // A LIVE fallback: an unreachable one errors before it can construct an answer, and this // test is about the answer's fields. - let be = WalletBackend::new(db, Arc::new(MockFallback::default()), WalletConfig::default()) - .with_chain_peer_tier_for_tests(peers_ahead_of_the_replica()); + let be = WalletBackend::new( + db, + Arc::new(MockFallback::default()), + WalletConfig::default(), + ) + .with_chain_peer_tier_for_tests(peers_ahead_of_the_replica()); let result = be .balance_for_address(&owned_address(), BalanceAsset::Xch) diff --git a/crates/dig-wallet/src/sage/sync_supervisor.rs b/crates/dig-wallet/src/sage/sync_supervisor.rs index ac308d94..de04a331 100644 --- a/crates/dig-wallet/src/sage/sync_supervisor.rs +++ b/crates/dig-wallet/src/sage/sync_supervisor.rs @@ -923,7 +923,10 @@ impl RefusalReason { /// writer to have contradicted, so an implementation that compared answers first would report a /// perfectly honest writer as a liar on every split — and the caller replaces a contradicting /// writer promptly, so that mistake is a re-dial on every split rather than a mere mislabel. -pub fn refusal(round: &CorroborationRound, writer_answer: Option) -> Option { +pub fn refusal( + round: &CorroborationRound, + writer_answer: Option, +) -> Option { let Some(agreed) = round.verdict.corroborated() else { return Some(RefusalReason::Undecided); }; @@ -1476,11 +1479,7 @@ impl Supervisor { /// it is a floor the writer cannot inflate. It is fixed for the session and never ratchets — /// refreshing it is exactly what [`SESSION_MAX_LIFETIME`] rotation already does, which is the /// one place rotation REDUCES exposure rather than raising it. - async fn trust_for_session( - &self, - session: &dyn SyncSession, - splits: &mut u32, - ) -> SessionTrust { + async fn trust_for_session(&self, session: &dyn SyncSession, splits: &mut u32) -> SessionTrust { let dialed = session.trust(); if dialed != PeerTrust::Discovered { return SessionTrust::elevated(match dialed { diff --git a/scripts/acceptance-wallet-balance.sh b/scripts/acceptance-wallet-balance.sh index 788eecc2..b9347525 100644 --- a/scripts/acceptance-wallet-balance.sh +++ b/scripts/acceptance-wallet-balance.sh @@ -8,7 +8,7 @@ # # 1. does the node hold Chia peers? (dig_ecosystem#2806) # 2. does it FOLLOW this address? (#2823 enrolment, #2848 app-side) -# 3. does the replica keep up once it has caught up? (#2851 froze for hours) +# 3. does the replica hold data it can date? (#2851 froze for hours; #2869 honesty) # 4. does a read actually ROUTE to the replica? (#2866 / #2234) # # Every one of those shipped with green unit tests while the wallet was unusable end to end. Gate 4 @@ -47,11 +47,15 @@ tip=$(field chia_peer_peak_height) # GATE 2 — addresses followed. A measured zero here means nothing was ever enrolled. [ "$watched" != "None" ] && [ "${watched:-0}" -ge 1 ] || fail 2 "the node follows no addresses (watched_addresses=$watched)" -# GATE 3 — the replica is keeping up, not merely once-caught-up. A frozen replica reported `synced` -# for three hours across a 312-block drift, so the phase alone is not the test: the DISTANCE is. -[ "$replica" != "None" ] && [ "$tip" != "None" ] || fail 3 "a height is unobservable (replica=$replica tip=$tip)" -behind=$(( tip - replica )) -[ "$behind" -le 50 ] || fail 3 "the replica is $behind blocks behind the chain tip" +# GATE 3 — the replica HAS DATA and can say what it is as of. +# +# This gate used to require the replica to be within 50 blocks of the tip, and that was wrong for the +# reason dig_ecosystem#2869 makes explicit: a behind replica MUST still serve, and it now says so +# honestly (`synced: false` WITH its real `peak_height`). Failing the run on the distance would fail +# the correct behaviour. So the distance is reported as CONTEXT below, never as a pass condition; +# what is actually required is that the replica has a height to answer as of at all. +[ "$replica" != "None" ] || fail 3 "the replica reports no height, so it has nothing to answer as of" +if [ "$tip" != "None" ]; then behind=$(( tip - replica )); else behind="unknown"; fi # GATE 4 — the read reaches the replica. This is the one that cannot be faked by a layer test. bal_json=$(dign wallet balance "$ADDRESS" --json 2>/dev/null) || fail 4 "the balance read failed" @@ -67,6 +71,16 @@ if [ "$source" != "db" ]; then fi fail 4 "the balance was answered by '$source'. This address is not among the ones the node follows, so the replica holds no coins for it — enrol it, or pass an address that is enrolled" fi -[ "$synced" = "True" ] || fail 4 "a db-tier answer reported synced=$synced; only a replica read may claim a synced view" +# A db answer must say what it is AS OF. `synced` is now measured rather than asserted, so +# `synced=False` is a legitimate answer from a behind replica and is NOT a failure — but a figure +# with no height attached is unusable either way. +peak=$(printf '%s' "$bal_json" | python -c "import json,sys;print(json.load(sys.stdin).get('peak_height'))") +[ "$peak" != "None" ] || fail 4 "a db-tier answer carried no peak_height, so nothing says what it is as of" + +# The falsehood dig_ecosystem#2869 removed, asserted directly: a replica far behind the tip may serve, +# but it may NEVER present its figure as current. +if [ "$behind" != "unknown" ] && [ "$behind" -gt 50 ] && [ "$synced" = "True" ]; then + fail 4 "the replica is $behind blocks behind and still reported synced=True; a stale figure was presented as settled" +fi -echo "PASS: peers=$peers watched=$watched behind=$behind source=$source synced=$synced" +echo "PASS: peers=$peers watched=$watched behind=$behind source=$source synced=$synced peak=$peak" From 10b0238427101346eb9da5fcbb69c6196c451a2d Mon Sep 17 00:00:00 2001 From: Michael Taylor Date: Thu, 13 Aug 2026 15:54:33 -0700 Subject: [PATCH 5/7] chore(release): 0.118.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Minor: `synced` on a `Source::Db` answer becomes a measured value, and the supervisor gains a public `RefusalReason` / `refusal` / `SessionTrust` surface. No API is removed and no wire field is added or removed, so this is compatible new capability rather than a break — but the MEANING of an existing `synced: true` narrows, and a consumer that requires it is affected, which is why it is not a patch. Refs dig_ecosystem#2868 Co-Authored-By: Claude From d81bc457203a5b68bcb58921eec40ade9ab5df45 Mon Sep 17 00:00:00 2001 From: Michael Taylor Date: Thu, 13 Aug 2026 19:30:36 -0700 Subject: [PATCH 6/7] fix(wallet): refuse to claim currency with no observable peer height MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `is_following` answers `true` on an unobservable peer tier by design: on `control.wallet.syncStatus` an absent second opinion is not an accusation. A money read delegating to it unnarrowed therefore paired `synced: true` with an arbitrarily old `peak_height` whenever no chain peer had announced a height — a freshly started node, or one with no reachable chain peer — resting that claim on the latched `initial_sync_complete` this change exists to stop trusting. `replica_answer_is_current` now requires an observable tier before delegating. `is_following` itself is untouched, so the two endpoints keep agreeing wherever a peer height exists. The figure is still SERVED with its real `peak_height`, labelled stale. Also re-anchors `an_incomplete_catch_up_never_serves_a_dated_zero_from_the_replica` to `replica_is_authoritative` — dig_ecosystem#2871 replaced `is_synced` at both production call sites feeding `route`, so the test had drifted off the predicate it describes — and couples the acceptance script's stale assertion to the unobservable-peer state its `behind != unknown` guard cannot see. Refs dig_ecosystem#2869 Co-Authored-By: Claude --- crates/dig-wallet/src/sage/rpc.rs | 101 ++++++++++++++++++++++++--- scripts/acceptance-wallet-balance.sh | 6 ++ 2 files changed, 98 insertions(+), 9 deletions(-) diff --git a/crates/dig-wallet/src/sage/rpc.rs b/crates/dig-wallet/src/sage/rpc.rs index f553f943..1890adb1 100644 --- a/crates/dig-wallet/src/sage/rpc.rs +++ b/crates/dig-wallet/src/sage/rpc.rs @@ -837,10 +837,21 @@ impl WalletBackend { /// /// It reuses [`super::sync_supervisor::is_following`] — the SAME predicate /// `control.wallet.syncStatus` reports its phase from — so a client cannot be told `synced` by - /// one endpoint and `syncing` by the other about the same moment. An unobservable peer tier is - /// never an accusation there and is not one here. + /// one endpoint and `syncing` by the other about the same moment. + /// + /// It narrows that predicate in ONE direction, and only here: `is_following` answers `true` on + /// an UNOBSERVABLE peer tier, because on a status endpoint an absent second opinion is not an + /// accusation. On a money read it is the opposite — with no peer height to compare against, + /// nothing has established that the replica's figure is current, and `synced: true` would then + /// rest on the latched `initial_sync_complete` this method exists to stop trusting. So an + /// unobservable tier answers `false`: the figure is still SERVED, with its real `peak_height`, + /// labelled stale. `is_following` itself is deliberately left alone so the two endpoints keep + /// agreeing wherever a peer height exists. async fn replica_answer_is_current(&self, peak_height: Option) -> bool { - super::sync_supervisor::is_following(peak_height, self.chain_peer_tier().await.peak_height) + let Some(peer_peak) = self.chain_peer_tier().await.peak_height else { + return false; + }; + super::sync_supervisor::is_following(peak_height, Some(peer_peak)) } /// The chain-sync supervisor's handle, if one is running. @@ -4565,7 +4576,8 @@ mod tests { Some(1), None, )])); - let be = WalletBackend::new(db, fb.clone(), WalletConfig::default()); + let be = WalletBackend::new(db, fb.clone(), WalletConfig::default()) + .with_chain_peer_tier_for_tests(peers_level_at(500)); let r = be .balance_for_address(&owned_address(), BalanceAsset::Xch) @@ -4708,7 +4720,8 @@ mod tests { db, Arc::new(MockFallback::default()), WalletConfig::default(), - ); + ) + .with_chain_peer_tier_for_tests(peers_level_at(500)); let r = be .balance_for_address(&owned_address(), BalanceAsset::Xch) @@ -4751,7 +4764,8 @@ mod tests { Arc::new(MockFallback::default()), WalletConfig::default(), ) - .with_watchlist(registry); + .with_watchlist(registry) + .with_chain_peer_tier_for_tests(peers_level_at(500)); let r = be .balance_for_address(&encode_address(&ph, "xch").unwrap(), BalanceAsset::Xch) @@ -5131,7 +5145,8 @@ mod tests { db, Arc::new(MockFallback::default()), WalletConfig::default(), - ); + ) + .with_chain_peer_tier_for_tests(peers_level_at(500)); let r = be .balance_for_address(&owned_address(), BalanceAsset::Xch) .await @@ -5791,7 +5806,8 @@ mod tests { Some(1), None, )])); - let be = WalletBackend::new(db, fb.clone(), WalletConfig::default()); + let be = WalletBackend::new(db, fb.clone(), WalletConfig::default()) + .with_chain_peer_tier_for_tests(peers_level_at(500)); let r = be .coins_for_address(&owned_address(), BalanceAsset::Xch) @@ -5902,6 +5918,19 @@ mod tests { const PEERS_AHEAD_BY: u32 = 530; /// A tier whose peers announced a peak `PEERS_AHEAD_BY` blocks past the replica's. + /// An OBSERVABLE peer tier level with a replica at `peak`. + /// + /// Tests whose subject is routing or coin content still read `synced`, and + /// [`WalletBackend::replica_answer_is_current`] refuses to claim currency with no peer height + /// to compare against. Without this the fixture would answer `synced: false` for a reason + /// unrelated to what those tests exist to pin. + fn peers_level_at(peak: u32) -> super::super::fallback::ChainPeerTier { + super::super::fallback::ChainPeerTier { + peer_count: Some(5), + peak_height: Some(peak), + } + } + fn peers_ahead_of_the_replica() -> super::super::fallback::ChainPeerTier { super::super::fallback::ChainPeerTier { peer_count: Some(5), @@ -5980,6 +6009,55 @@ mod tests { assert_eq!(result.peak_height, Some(REPLICA_PEAK)); } + /// **Proves:** an UNOBSERVABLE peer tier is not a licence to claim currency — a node with no + /// chain peer that has announced a height serves its figure labelled stale. + /// + /// This is the state a freshly-started node sits in, and the one a node with no reachable chain + /// peer sits in indefinitely. [`super::sync_supervisor::is_following`] answers `true` there by + /// design (an absent second opinion is not an accusation on a status endpoint), so a money read + /// delegating to it unnarrowed pairs `synced: true` with an arbitrarily old `peak_height` — the + /// stale-presented-as-current claim this PR exists to remove. + /// + /// FIXTURE DESIGN — `peak_height: None` is what makes the tier unobservable, and it is the only + /// axis varied from [`a_replica_level_with_its_peers_still_reports_synced`], which stays green + /// as the honest control. The replica is deliberately CAUGHT UP (`initial_sync_complete`, a + /// present peak, a real coin), so nothing but the missing peer height can explain a `false`; + /// asserting the balance and the peak alongside pins "stale but served" over "withheld". + #[tokio::test] + async fn an_unobservable_peer_tier_is_never_reported_as_current() { + let db = db_with_owned_derivation(true, Some(REPLICA_PEAK)).await; + db.upsert_coin(&coin_at_ph( + "aa", + &owned_ph(), + 1_599_179_999_973, + Some(1), + None, + )) + .await + .unwrap(); + let be = WalletBackend::new(db, Arc::new(EmptyFallback), WalletConfig::default()) + .with_chain_peer_tier_for_tests(super::super::fallback::ChainPeerTier { + peer_count: None, + peak_height: None, + }); + + let result = be + .balance_for_address(&owned_address(), BalanceAsset::Xch) + .await + .unwrap(); + assert_eq!(result.source, Source::Db, "the replica stopped serving"); + assert_eq!(result.balance, 1_599_179_999_973, "the figure was withheld"); + assert_eq!( + result.peak_height, + Some(REPLICA_PEAK), + "a stale answer must still say WHAT it is as of" + ); + assert!( + !result.synced, + "a figure no peer height could corroborate was reported as current" + ); + } + /// **Proves:** the coin read makes the SAME claim as the balance read about the same replica. /// /// They are the same answer reduced differently, and a caller building a spend reads this one. @@ -6060,8 +6138,13 @@ mod tests { "the fixture must carry a peak; without one it proves nothing" ); + // Anchored to `replica_is_authoritative`, NOT to `db.is_synced()`: dig_ecosystem#2871 + // replaced the latter at both production call sites feeding `route`, so a test still asking + // `is_synced` would describe a predicate no money read consults — green even if + // `replica_is_authoritative` started trusting a never-caught-up replica. + let be = WalletBackend::new(db, Arc::new(EmptyFallback), WalletConfig::default()); assert_eq!( - routing::route(db.is_synced().await.unwrap(), true), + routing::route(be.replica_is_authoritative().await.unwrap(), true), Source::Fallback, "a replica with a peak but no completed catch-up was served as authoritative; it holds \ no coins, so that answer is a dated zero for a funded wallet" diff --git a/scripts/acceptance-wallet-balance.sh b/scripts/acceptance-wallet-balance.sh index b9347525..7004ea63 100644 --- a/scripts/acceptance-wallet-balance.sh +++ b/scripts/acceptance-wallet-balance.sh @@ -82,5 +82,11 @@ peak=$(printf '%s' "$bal_json" | python -c "import json,sys;print(json.load(sys. if [ "$behind" != "unknown" ] && [ "$behind" -gt 50 ] && [ "$synced" = "True" ]; then fail 4 "the replica is $behind blocks behind and still reported synced=True; a stale figure was presented as settled" fi +# The same falsehood in the state the check above CANNOT see. `behind` is "unknown" exactly when no +# peer height is observable — which is precisely when nothing has corroborated the replica's figure, +# so the guard would otherwise skip in the one case the node is most able to lie about. +if [ "$behind" = "unknown" ] && [ "$synced" = "True" ]; then + fail 4 "no peer height is observable, yet the replica reported synced=True; nothing corroborated that figure" +fi echo "PASS: peers=$peers watched=$watched behind=$behind source=$source synced=$synced peak=$peak" From 5844a55a23cdec1175968e2c162040d72482112f Mon Sep 17 00:00:00 2001 From: Michael Taylor Date: Thu, 13 Aug 2026 19:33:02 -0700 Subject: [PATCH 7/7] docs(spec): state that a money read refuses currency without a peer height Co-Authored-By: Claude --- SPEC.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 04b436b3..5b90d64a 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1505,7 +1505,7 @@ lowercase 64-hex; a capsule reference is `storeId:rootHash`. Malformed refs yiel | `control.hostedStores.status` | `store` = `storeId[:rootHash]` | `store_id`, `pinned`, `capsule_count`, `total_bytes`, `capsules[]` | | `control.sync.status` | — | `available` (always `true` — the chunked capsule download needs no identity), `method: "chunked-capsule-download-with-section-21-clone-fallback"`, `identity_loaded`, `pinned_total`, `pinned_synced`, `whole_store_trigger_supported` (`true` — a store id alone is enough) | | `control.sync.trigger` | `store` = `storeId[:rootHash]`, or `store_id` [+ `root`] — the root is OPTIONAL; without one the node resolves the store's CHAIN-ANCHORED tip and syncs that generation | `status: "synced"`, `root`, `size_bytes`, `served_root` | -| `control.wallet.balance` | `address` (bech32m string), `asset` (`"xch"` \| `"dig"`, default `"xch"`) | `balance` (confirmed, spendable — JSON NUMBER, u64 base units), `pending` (unspent + unconfirmed — JSON NUMBER, u64 base units), `source` (`"db"` \| `"fallback"` — which tier produced the figure, §18.7b), `synced` (bool), `peak_height` (`u32` or `null`). Matches `dig-node-control-interface` 0.3.0's `WalletBalanceResult { balance: u64, pending: u64, .. }` and dig-app's `BalanceResponse { balance: u64 }` — a Rust-to-Rust numeric contract, never a decimal string. The wallet backend tracks the base-unit total as `u128` (headroom for summed intermediate math); the wire boundary saturating-casts to `u64` (a single address's balance can never exceed `u64::MAX` mojos, ~18.4M XCH). READ-ONLY chain read of a PUBLIC address (no seed/signing key). Reuses the B.6 sync-state routing: the local DB when the address is the wallet's own and the DB is synced, else the coinset fallback. Per §18.7b, `source`/`synced`/`peak_height` describe the TIER that answered: a `"db"` answer reports the node's own peak and reports `synced: true` only while the replica is FOLLOWING the chain, so a behind-but-once-synced replica answers `synced: false` WITH its real `peak_height` rather than presenting a stale figure as current; a `"fallback"` answer reports `synced: false` and `peak_height: null`. This is an OPEN read (`is_open_control_read`, no token); the cheap local-DB fast path is unbounded, but the EXPENSIVE coinset-fallback leg is subject to a GLOBAL token-bucket rate bound (defense-in-depth against an open-read amplification/oracle sweep — #1957): a burst of arbitrary-address fallback reads beyond the bound is refused with `WALLET_RATE_LIMITED` (§10), while any single honest read (DB fast path or one fallback) always succeeds. `$DIG` scopes by the canonical CAT asset id `digstore_chain::dig::DIG_ASSET_ID`. A synced empty address is a SUCCESS `{balance:0, synced:true}`, never an error (and `synced` there means MEASURED-current, never merely eligible); the read-failure shapes are DISTINCT errors `WALLET_NO_CHAIN_SOURCE`/`WALLET_NOT_SYNCED`/`WALLET_READ_FAILED`/`WALLET_RATE_LIMITED` (§10), never a fabricated `0`. `INVALID_PARAMS` on a missing/malformed `address` or a bad `asset`. | +| `control.wallet.balance` | `address` (bech32m string), `asset` (`"xch"` \| `"dig"`, default `"xch"`) | `balance` (confirmed, spendable — JSON NUMBER, u64 base units), `pending` (unspent + unconfirmed — JSON NUMBER, u64 base units), `source` (`"db"` \| `"fallback"` — which tier produced the figure, §18.7b), `synced` (bool), `peak_height` (`u32` or `null`). Matches `dig-node-control-interface` 0.3.0's `WalletBalanceResult { balance: u64, pending: u64, .. }` and dig-app's `BalanceResponse { balance: u64 }` — a Rust-to-Rust numeric contract, never a decimal string. The wallet backend tracks the base-unit total as `u128` (headroom for summed intermediate math); the wire boundary saturating-casts to `u64` (a single address's balance can never exceed `u64::MAX` mojos, ~18.4M XCH). READ-ONLY chain read of a PUBLIC address (no seed/signing key). Reuses the B.6 sync-state routing: the local DB when the address is the wallet's own and the DB is synced, else the coinset fallback. Per §18.7b, `source`/`synced`/`peak_height` describe the TIER that answered: a `"db"` answer reports the node's own peak and reports `synced: true` only while the replica is FOLLOWING the chain, so a behind-but-once-synced replica answers `synced: false` WITH its real `peak_height` rather than presenting a stale figure as current; a tier with NO observable peer height also answers `synced: false`, because nothing corroborated the figure (§18.7b); a `"fallback"` answer reports `synced: false` and `peak_height: null`. This is an OPEN read (`is_open_control_read`, no token); the cheap local-DB fast path is unbounded, but the EXPENSIVE coinset-fallback leg is subject to a GLOBAL token-bucket rate bound (defense-in-depth against an open-read amplification/oracle sweep — #1957): a burst of arbitrary-address fallback reads beyond the bound is refused with `WALLET_RATE_LIMITED` (§10), while any single honest read (DB fast path or one fallback) always succeeds. `$DIG` scopes by the canonical CAT asset id `digstore_chain::dig::DIG_ASSET_ID`. A synced empty address is a SUCCESS `{balance:0, synced:true}`, never an error (and `synced` there means MEASURED-current, never merely eligible); the read-failure shapes are DISTINCT errors `WALLET_NO_CHAIN_SOURCE`/`WALLET_NOT_SYNCED`/`WALLET_READ_FAILED`/`WALLET_RATE_LIMITED` (§10), never a fabricated `0`. `INVALID_PARAMS` on a missing/malformed `address` or a bad `asset`. | | `control.wallet.coins` | `address` (bech32m string), `asset` (`"xch"` \| `"dig"`, default `"xch"`) | `coins` (array of `{coin_id, asset, amount, parent_coin_info, puzzle_hash, created_height, spent_height}`; all hashes lowercase 64-hex unprefixed, `amount` a JSON NUMBER in base units), `source`, `synced`, `peak_height` — the tier fields carrying exactly their `control.wallet.balance` meanings (§18.7b). The UNSPENT coins at the address for the asset, i.e. the read a caller building a spend needs; a balance is this read reduced to a sum, which is why the two take identical params. Coins seen only in the mempool are INCLUDED with `created_height: null`, so the caller decides what is spendable for its purpose rather than the node hiding one. `coins: []` MUST mean a chain WAS consulted and the address holds nothing; every way of failing to consult one is a DISTINCT error (`WALLET_NO_CHAIN_SOURCE`/`WALLET_NOT_SYNCED`/`WALLET_READ_FAILED`/`WALLET_RATE_LIMITED`, §10), NEVER an empty list — an empty list would tell a holder of funds that they hold none, and a spend built on it refuses with an untrue shortfall. OPEN read, same global fallback rate bound as the balance. `INVALID_PARAMS` on a missing/malformed `address` or a bad `asset`. | | `control.wallet.coinById` | `coin_id` (64 lowercase-hex, an optional `0x` prefix TOLERATED and normalized away) | `coin` (`{coin_id, asset, amount, parent_coin_info, puzzle_hash, created_height, spent_height}` or `null`), `source`, `synced`, `peak_height` — the tier fields carrying exactly their `control.wallet.balance` meanings (§18.7b); `source` is always `"fallback"`, `synced` always `false` and `peak_height` always `null`, because the answer never comes from the local replica (a miss there means "this node does not watch that coin", which is NOT absence). ONE coin by its own id, SPENT OR UNSPENT — the read a caller polling a spend needs and `control.wallet.coins` structurally cannot give: a created DID coin sits at nobody's wallet address, and a spent funding coin is gone from every unspent list. `asset` in the record is ALWAYS `null`: a coin id alone does not reveal whether a coin is XCH, a CAT or a singleton — that needs the puzzle, which this read never inspects — so naming one would assert a classification the node did not verify. A returned record MUST be bound to the id asked for: a coin id is self-certifying (`SHA256(parent ‖ puzzle_hash ‖ amount)`), so a source that answers with a DIFFERENT coin is a `WALLET_READ_FAILED` (§10) — never that coin's record, and never `coin: null`. `coin: null` MUST mean a chain source ANSWERED and reported no such coin; every way of failing to get an answer is a DISTINCT error (`WALLET_NO_CHAIN_SOURCE`/`WALLET_READ_FAILED`/`WALLET_RATE_LIMITED`, §10), NEVER a `null` — a `null` for an outage would tell a caller polling a mint that its coin does not exist, so a pending mint reads as awaiting forever. A caller MUST treat `null` as "not seen yet" and keep polling, not as "never happened". OPEN read (no token), same global fallback rate bound as the balance. `INVALID_PARAMS` on a missing/malformed `coin_id`, refused BEFORE any network call — an unanswerable question and a chain that answered "no" must never wear the same shape; the well-formedness rule is `dig-node-control-interface`'s own `WalletCoinByIdParams::validated()`, consumed rather than restated. | | `control.wallet.coinSpend` | `coin_id` (64 lowercase-hex, an optional `0x` prefix TOLERATED and normalized away) | `spend` (`{coin, puzzle_reveal, solution}` or `null`), `source`, `synced`, `peak_height` -- the tier fields carrying exactly their `control.wallet.coinById` meanings and the same always-`"fallback"` / always-`false` / always-`null` values, for the same reason. THE SPEND THAT SPENT ONE COIN, named by that coin's own id (a spend has no id of its own on chain). A coin record carries a puzzle HASH and says only that a coin is gone; the puzzle REVEAL and the solution exist only here, and they are what a caller reconstructing a lineage -- following a dig-profile's DID singleton forward -- needs. `coin` is the full record shape `control.wallet.coinById` returns, with `asset` ALWAYS `null` (this read classifies nothing) and `spent_height` ALWAYS non-null (a spend of a coin nothing calls spent is a contradiction; the node MUST fail closed rather than emit one). The node MUST verify that `puzzle_reveal` tree-hashes to `coin.puzzle_hash` and MUST refuse -- `WALLET_READ_FAILED` (§10) -- when it does not or will not parse: the reveal comes from an unauthenticated peer, a puzzle hash IS the reveal's CLVM tree hash, so the lie is locally detectable and a caller would otherwise curry a forged program into the spend it signs. The returned spend MUST be bound to the id asked for, by the same self-certifying coin-id recomputation `control.wallet.coinById` requires. `spend: null` MUST mean a chain source ANSWERED and holds no spend of that coin -- it is UNSPENT, or unknown; distinguishing those two is `control.wallet.coinById`'s job. Every way of failing to get an answer is a DISTINCT error, NEVER `null`: a caller walking a lineage reads "no spend" as *this is the tip* and stops, so a failure disguised as absence yields a spend built against a superseded singleton, and a mint poll reads it as "my funding coin is still there" and funds the same mint twice. OPEN read (no token), same global fallback rate bound. `INVALID_PARAMS` on a missing/malformed `coin_id`, refused BEFORE any network call; the rule is `dig-node-control-interface`'s own `WalletCoinSpendParams::validated()`, consumed rather than restated. | @@ -4386,6 +4386,16 @@ answered, and MUST derive every freshness field from that tier. withhold it or blank the peak: `peak_height` is what makes a stale answer usable rather than merely suspect. + **A money read narrows `is_following` in exactly ONE direction: an UNOBSERVABLE peer tier — no + chain peer has announced a height — MUST answer `synced: false`.** `is_following` itself answers + `true` there, and MUST continue to, because on `control.wallet.syncStatus` an absent second + opinion is not an accusation against the replica. On a money read it is the opposite: with no peer + height to compare against, nothing has established that the figure is current, so `synced: true` + would rest on the latched `initial_sync_complete` this rule exists to stop trusting — the state a + freshly started node, or one with no reachable chain peer, sits in. The figure is still SERVED + with its real `peak_height`, per the paragraph above. Wherever a peer height DOES exist the two + endpoints apply the identical test, so they still cannot disagree about the same moment. + A `"fallback"` answer reports `synced: false` and `peak_height: null`, **regardless of the local DB's state** — the DB neither produced that figure nor bounds its freshness, so its flag and peak say nothing about it. Implementations MUST NOT read those two fields outside the tier decision.