Fix funding-payment reclassification downgrade and deep-reorg duplication - #962
Fix funding-payment reclassification downgrade and deep-reorg duplication#962jkczyz wants to merge 9 commits into
Conversation
|
👋 Thanks for assigning @joostjager as a reviewer! |
| // below (gated on `Pending`) that graduation removed; without it a graduated payment would | ||
| // be left `Succeeded` with an `Unconfirmed` kind and no way to re-graduate. | ||
| if matches!(confirmation_status, ConfirmationStatus::Unconfirmed) { | ||
| payment.status = PaymentStatus::Pending; |
There was a problem hiding this comment.
As mentioned over at #888 (comment) I'm not sure we do this, as our base assumption is that anything beyond ANTI_REORG_DELAY can't be reorged anyways, hence why we have the entries only graduate after ANTI_REORG_DELAY? As mentioned in that comment, maybe it would be easier to fail early if the user tries to bump a confirmed splice?
There was a problem hiding this comment.
As mentioned over at #888 (comment) I'm not sure we do this, as our base assumption is that anything beyond
ANTI_REORG_DELAYcan't be reorged anyways, hence why we have the entries only graduate afterANTI_REORG_DELAY?
Sorry, you're right. This commit isn't needed. Dropping it will address the codex issues, too.
As mentioned in that comment, maybe it would be easier to fail early if the user tries to bump a confirmed splice?
Yeah, that should be covered as we check the tx_type in bump_fee_rbf. Or did you mean when using bump_channel_funding_fee we should error when RBF is possible according to LDK (i.e., the splice isn't locked yet), but we've already reached ANTI_REORG_DELAY confirmations?
There was a problem hiding this comment.
Right, IIUC we could avoid the race if we just don't proceed when we previously had reached ANTI_REORG_DELAY already?
There was a problem hiding this comment.
Yeah, though we should use ChannelDetails::splice_details` from https://git.rust-bitcoin.org/lightningdevkit/rust-lightning/pulls/4687 rather than looking at the pending payment store.
There was a problem hiding this comment.
Yeah, though we should use ChannelDetails::splice_details` from https://git.rust-bitcoin.org/lightningdevkit/rust-lightning/pulls/4687 rather than looking at the pending payment store.
Alright, so that probably means we want to wait for the backport of that PR to land and the API to become available?
There was a problem hiding this comment.
Yeah, and it looks like there are other splicing backports that need to happen first.
| // `find_payment_by_txid`'s payment-store fallback. Revert it like the | ||
| // `TxUnconfirmed`/`TxDropped` arms instead of mirroring a non-`Pending` record | ||
| // into the pending store, which graduation's pending-only scan would reject. | ||
| if payment.status != PaymentStatus::Pending |
There was a problem hiding this comment.
Codex:
- P1 /home/tnull/workspace/ldk-node-pr-962/src/wallet/mod.rs:410: TxReplaced uses BDK’s replaced txid as if it were the active unconfirmed tx. For a graduated funding payment, this stamps the old replaced txid and continues without recording conflicts; the later replacement TxUnconfirmed/
TxConfirmed will not map back and can create the duplicate this PR is trying to prevent.
There was a problem hiding this comment.
Codex:
- P1 /home/tnull/workspace/ldk-node-pr-962/src/wallet/mod.rs:265: TxConfirmed computes the depth-aware payment_status, but the funding short-circuit ignores it. A graduated funding payment re-confirmed in a new shallow block after a reorg can stay Succeeded, skip pending-store recreation,
and bypass ANTI_REORG_DELAY.
| self.runtime.block_on(self.pending_payment_store.insert_or_update(pending))?; | ||
| } | ||
| Ok(true) | ||
| } |
There was a problem hiding this comment.
Codex:
- P2 /home/tnull/workspace/ldk-node-pr-962/src/wallet/mod.rs:1551: re-created pending entries after graduation use an empty candidate list. If a different RBF candidate confirms after a deep reorg, pending.candidate(event_txid) cannot restore the correct amount/fee, so the payment can report
stale figures.
There was a problem hiding this comment.
Hmm, it seems the wallet sync/broadcaster race will also be a problem for #448, as there we'd then emit OnchainPayment{Successful,Received} events for transactions that then will be reclassified as channel-related (for which we'd usually not emit these events).
@jkczyz Any idea how we could avoid this class of error entirely? Or maybe it won't be an issue in practice if we stick to emitting the event only after ANTI_REORG_DELAY conf I guess?
This won't work for the restart case if the node is offline during confirmation for more than There are three realistic options, which can be combined: 1. Just wait for 2. Check channel state directly when about to emit. Instead of trusting the 3. Label more transaction types at broadcast. Today only funding txs get labeled; Weaker ideas I'd rule out: forcing the wallet sync to wait for the broadcast queue to My take: 3 is needed regardless, 2 is what actually eliminates the class, and 1 is a |
Hmm, so 3 is already done in #791 (so this comment seems somewhat stale), in the current version of #448 we already do 1. But, it seems 2 / the restart issue might be a good reason to move forward with #946 after all? |
Yeah, though for splicing most data can still live in the pending payment store. The channel store would only need funding outpoints to check. When we introduce batching, we wouldn't want to duplicate all that data across each channel in the storage. |
tnull
left a comment
There was a problem hiding this comment.
This needs a rebase by now.
Also, some additional claude comments:
Conflicts and candidates are dropped when reverting a graduated payment. The revert path re-creates the pending entry via create_pending_payment_from_tx(payment, Vec::new()) — empty conflicting_txids and empty candidates (the graduation removal already discarded the originals).
Consequence: if, after the deep reorg, a different RBF candidate confirms than the stamped one, an earlier/middle candidate's txid no longer maps to the payment (duplicate record — the very bug class this PR fixes), and even for the first candidate the confirmed-candidate figures can't be
re-stamped since pending.candidate(event_txid) finds nothing. In practice LDK's re-broadcast of the candidate re-runs classify_interactive_funding, whose merge path (commit 1) restores the full candidate history, so this likely self-heals — but that's an implicit dependency worth a comment
or an upstream question. Similarly, the graduated-funding TxReplaced path continues without recording the event's conflict txids.Residual TOCTOU in persist_funding_payment (src/wallet/mod.rs:1418). contains_key and the subsequent insert aren't atomic; a wallet sync landing in between would make the fresh-insert path do a full insert_or_update merge that could still clobber confirmation state. The window is tiny and
strictly better than before (the old code clobbered unconditionally), so a nit only.Documented invariant worth upstream confirmation. The funding_reclassification doc comment leans on "LDK only re-broadcasts the active/confirmed funding candidate" to justify unconditionally overwriting txid/amount/fee. If LDK ever rebroadcasts a non-confirmed candidate for an
already-confirmed record, the stamped figures would be silently replaced. Fine to rely on, but it's the kind of cross-crate invariant a reviewer on the LDK side should ack.
| // merges into it), so the first match is unambiguous. | ||
| if let Some(funding) = self | ||
| .payment_store | ||
| .list_filter(|p| { |
There was a problem hiding this comment.
I really don't think we can do this - this would scan all payment store entries constantly, which is a no-go even if all of them live in memory. And going forward we'll also want to only keep a cache of payments in memory while most of them live just in the KVStore.
To make this more efficient we probably need a secondary index, similar to what we'll do in #948.
There was a problem hiding this comment.
I believe this is no longer a problem since the commit adding this is now dropped.
f01e83e to
46096b0
Compare
Funding broadcasts are classified into payment records off the broadcaster's queue, which can run after wallet sync has already recorded the transaction -- for instance when the counterparty's broadcast of the funding transaction is observed by wallet sync first. In that case the classification overwrote a record wallet sync had already advanced, downgrading a confirmed or graduated funding payment back to unconfirmed/pending. Merge only the classification and our contribution figures into an existing record, leaving the confirmation state that the wallet-sync events own in place. Raised by Codex in the review of lightningdevkit#888. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Push is just a rebase plus dropping a commit as per #962 (comment). I have some of the other issues addressed locally but need to verify the work still. |
Alright, please re-request review when it's ready! |
…ce check Co-Authored-By: Claude <noreply@anthropic.com>
…fication Co-Authored-By: Claude <noreply@anthropic.com>
46096b0 to
0e44736
Compare
|
PTAL
Dropping the earlier commit fixes this.
Added fixups for these two. |
tnull
left a comment
There was a problem hiding this comment.
Still have to take a closer look.
In general, I do wonder if with the current design we'll really be able to whack-a-mole all edge cases here :(
| // confirmed, which need not be the last one broadcast. An update that doesn't assert the | ||
| // confirmation state was built without knowing it — e.g. a late funding classification | ||
| // whose candidate lost to the counterparty's broadcast — so it must not move them. | ||
| let keep_confirmed_figures = update.confirmation_status.is_none() |
There was a problem hiding this comment.
Codex:
- P1 — /home/tnull/worktrees/ldk-node/pr-962-latest-20260730/src/payment/store.rs:250: keep_confirmed_figures assumes every confirmed record already contains the correct funding-candidate figures. In the race this PR handles, wallet sync may have created a generic tx_type: None record using
BDK-derived figures, which the code itself says cannot represent our contribution to shared funding. Late classification now preserves those incorrect figures even though its candidate history contains the correct values. It should select the candidate matching the stored confirmed txid,
rather than unconditionally preserving generic figures.
There was a problem hiding this comment.
Added a fixup addressing this for now.
| let pending = PendingPaymentDetails::new(details, Vec::new(), candidates); | ||
| self.pending_payment_store.update_or_insert(pending_update, pending).await?; | ||
| }, | ||
| DataStoreUpdateOrInsertResult::Updated | DataStoreUpdateOrInsertResult::Unchanged => { |
There was a problem hiding this comment.
Codex:
- P2 — /home/tnull/worktrees/ldk-node/pr-962-latest-20260730/src/wallet/mod.rs:1428: an absent pending entry is treated as necessarily graduated. If the payment-store write succeeds but the pending-store write fails—or the node stops between them—a retry takes the Updated/Unchanged branch
and silently ignores NotFound. Main’s unconditional insert_or_update repaired this state. For an RBF splice, the missing pending index prevents find_payment_by_txid from mapping the replacement txid to the stable payment ID, potentially producing a duplicate generic payment. A missing
entry should be recreated when the authoritative payment is still Pending.
There was a problem hiding this comment.
Added a fixup addressing this for now.
Yeah, I seem to be running into similar problems in the dependent PR (#930). What approach were you thinking? Adding a secondary index (txid -> payment_id) and maybe consolidating the two stores? |
The keep-confirmed-figures guard overshot: when wallet sync recorded the confirmation first, the record carries the wallet's own view of amount/fee, which cannot represent our contribution to a shared funding output, and the guard then discarded the late classification's correct contribution-derived figures along with the losing-candidate updates it was meant to block. Let an update that names the confirmed txid move the figures — it describes the very candidate that confirmed — and have classification build its update from the candidate matching an already-confirmed record, mirroring what apply_funding_status_update reports when confirmation arrives after classification. The txid comparison happens under the store's mutation lock at apply time, so the unlocked snapshot read cannot misapply figures. Generated with assistance from Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Classification treated an absent pending entry as proof the payment had graduated and only merged into existing entries. But a crash or failed write between the payment-store and pending-store writes leaves a Pending record with no index entry, and that state was never repaired: the payment could no longer graduate (graduation iterates the pending store) and its candidate txids could no longer be mapped back to the record, which for an RBF splice invites a duplicate generic payment. Decide by the post-write payment status instead: while the record is still Pending, insert the missing entry (embedding the post-write record, so a confirmation wallet sync already mirrored keeps driving graduation); once it advanced beyond Pending, keep treating absence as graduated. A graduated payment is never Pending, so the no-reindex rule is preserved by the status gate itself. The repaired state is not constructible in a test: it requires a failure injected between the two store writes, and no such seam exists. The store primitives the decision rests on are unit-tested. Generated with assistance from Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| // The inserted entry embeds the post-write record rather than the fresh details, so a | ||
| // confirmation wallet sync already recorded keeps driving graduation. | ||
| let pending = PendingPaymentDetails::new(recorded, Vec::new(), candidates); | ||
| self.pending_payment_store.update_or_insert(pending_update, pending).await?; |
There was a problem hiding this comment.
AI drive-by
P1: This still races with graduation. recorded is an unlocked snapshot taken at line 1429. After it observes Pending, ChainTipChanged can update the authoritative payment to Succeeded and remove the pending entry. This update_or_insert then sees the entry absent and recreates it from the stale Pending snapshot. If that snapshot was also Unconfirmed, later chain-tip processing can repeatedly rebroadcast an already-graduated transaction. The two stores have independent mutation locks, so the status decision and index insertion need one cross-store critical section or transactional record.
There was a problem hiding this comment.
Moved the mutate method from #930 here to address this, but updated it to clone. Still worth considering options for more robust handling as mentioned here: #962 (comment)
The check that only Pending payments enter the pending index read the payment store before taking the pending store's lock. Graduation could write Succeeded and remove the index entry between the check and the write, and the stale check would then re-create an entry for the graduated payment. The next chain tip re-graduates it from the entry's stale embedded copy — or, if that copy was still unconfirmed, keeps rebroadcasting an already-confirmed transaction on every tip. Move the decision into the pending store's critical section via a new DataStore::mutate that reads, transforms, and persists an entry under one hold of the mutation lock. Re-reading the payment's status there is race-free because graduation writes Succeeded before removing the entry: a read that still observes Pending precedes the removal, which then also deletes anything inserted here. The closure runs with the in-memory map lock released, so it may read other stores without ordering the stores' map locks against each other. The race spans a few instructions between two store writes and no seam exists to schedule a graduation inside it, so no test exercises it; the new unit tests cover the mutate primitive itself. Generated with assistance from Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Well, it seems to me that the fundamental issue is the chosen approach of classifying at time of broadcast which can happen before or after the wallet sync, i.e., after we actually see the transaction. So fixing it would be reconsidering that model, i.e., offer an LDK interface along the lines of |
| let update = funding_reclassification_update( | ||
| details.clone(), | ||
| &candidates, | ||
| self.payment_store.get(&details.id).as_ref(), |
There was a problem hiding this comment.
Codex:
[P1] Build the reclassification from the locked payment state — /home/tnull/worktrees/ldk-node/pr-962-latest-20260805/src/wallet/mod.rs:1411
payment_store.get() may return pre-write state while wallet sync is persisting a confirmation. Classification then waits for the mutation lock but applies an update derived from that stale snapshot. If candidate A confirms while candidate B is classified, the update still names B; the
confirmed-record guard consequently preserves wallet-derived figures instead of applying A’s contribution figures. Candidate selection needs to happen inside the same locked transformation as the write.
Maybe we can make the update_or_insert below also a mutate and move the funding_reclassification_update call into it?
| // is ordered before the removal, which then also deletes anything inserted here. A | ||
| // status read taken before this write goes stale when graduation lands in between, and | ||
| // would re-index the graduated payment. | ||
| self.pending_payment_store |
There was a problem hiding this comment.
Codex:
[P1] Reconcile confirmations during pending-index persistence — /home/tnull/worktrees/ldk-node/pr-962-latest-20260805/src/wallet/mod.rs:1431
After the payment-store write, candidate history is not visible until pending_payment_store.mutate() finishes persisting. A concurrent confirmation can therefore see no candidates, stamp confirmed candidate A with active candidate B’s figures, and create the pending entry. The
classification path subsequently adds the candidates but never repairs the authoritative payment record, leaving the wrong amount/fee permanently. Candidate history must be visible before confirmation handling, or the payment record must be reconciled afterward.
Hmm, maybe for now we need to introduce a funding_payment_update_lock: tokio::sync::Mutex<()> that we can hold across both payment store and pending payment store updates to ensure they are always in-sync?
persist_funding_payment chose which candidate's figures to merge by reading the payment record before taking the store's mutation lock. A wallet sync confirming a candidate between that read and the write left the choice stale: the update still named the actively-broadcast candidate, so the confirmed-figures guard rightly refused it, and the record kept figures no classification derived — wrong for a shared funding output, and frozen permanently if the payment graduated before another event for the confirmed candidate arrived. The whole decision — insert or merge, and which candidate's figures the record's state makes authoritative — now runs inside the store's critical section, where a concurrent confirmation is either fully visible and substituted, or lands after this write and reads the candidate history itself. The race has no test seam (nothing can interpose between the read and the write), so it is not exercised by a test; the decision's single-threaded behavior is unchanged and remains covered by the existing tests. update_or_insert loses its only caller and is removed. Fixes the first finding in lightningdevkit#962 (comment) Implemented with the assistance of AI tooling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
apply_funding_status_update fetched the payment record, checked that it was a classified funding payment, merged in the new confirmation status, and wrote the result back as separate store operations. A classification racing in between -- merging tx_type and the contribution-derived figures into the record -- would be overwritten by the stale snapshot. Perform the check and the merge under the payment store's mutation lock so no write can interleave, and skip persisting when the merge changes nothing. Implemented with the assistance of AI tooling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Classification writes the payment record and the pending entry carrying the candidate history as two store operations. A funding confirmation processed between them saw the record already classified but the candidate history still absent, so it stamped the confirmed candidate's txid with a stale snapshot's figures -- and once the candidates landed, nothing revisited the payment record to repair them, leaving the wrong amount/fee to graduate with the payment. A wallet-level lock now serializes classification's two-store write pair against the funding-confirmation handling, so a confirmation either runs before the record is classified or sees the full candidate history. Writers touching a single store (e.g. graduation) are unaffected; the per-store gates continue to cover them. The race needs a confirmation interposed between two writes of one classification call, which no test can arrange; the lock is uncontended in the single-writer paths the existing tests exercise. Fixes the finding in lightningdevkit#962 (comment) Implemented with the assistance of AI tooling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Comments addressed. Please throw your agents at it again. |
Two independent fixes Codex raised in the review of #888: