Skip to content

fix(storage): make pruning crash-consistent - #3904

Open
DicksonWu654 wants to merge 1 commit into
sei-protocol:mainfrom
DicksonWu654:agent/storage-crash-consistency
Open

fix(storage): make pruning crash-consistent#3904
DicksonWu654 wants to merge 1 commit into
sei-protocol:mainfrom
DicksonWu654:agent/storage-crash-consistency

Conversation

@DicksonWu654

Copy link
Copy Markdown

Summary

Make storage pruning decisions crash-consistent across BlockDB, ReceiptDB, and StateWAL.

This change:

  • measures GC rollback windows from the highest crash-recoverable block rather than the newest buffered or written block;
  • caps direct pruning at each store's durable head, including BlockDB's newest durable QC cohort;
  • persists and recovers a durable ReceiptDB head after Litt flushes, without promoting an index-only suffix inherited from an earlier crash;
  • commits ReceiptDB's tag-index range tombstone and m:earliest floor atomically with Sync: true before publishing the in-memory floor;
  • adds regressions for pre-flush heads, concurrent flush/write boundaries, restart recovery, and receipt-floor commit ordering.

Why

BlockDB writes, StateWAL appends, and Litt receipt bodies are not crash-durable until their respective flush completes. The collector-facing implementations previously calculated retention from newer volatile cursors. A prune followed by a process crash could therefore lose the suffix used to justify the prune and leave less recoverable history than intended.

Receipt pruning also issued the Pebble range tombstone and retention-floor metadata as separate non-sync commits. Those form one logical visibility transition: a crash could retain only one side, and publishing a non-durable floor could release Litt bodies before the floor itself survived restart.

The fixes are conservative: failures, marker loss, or overlapping writes leave the reported head/floor behind and retain extra data rather than over-pruning.

Impact

This is storage crash-consistency hardening. It is not presented as a remotely exploitable security vulnerability or bounty submission. The shared storage garbage collector is not yet wired into production at this revision; the ReceiptDB local-pruning path benefits immediately when the Litt index backend and retention are enabled.

Validation

  • go test ./sei-db/db_engine/pebbledb ./sei-db/ledger_db/receipt ./sei-db/management/gc ./sei-db/state_db/statewal ./sei-db/ledger_db/block/littblock -count=1
  • same package set with -race
  • go test ./x/evm/keeper ./sei-tendermint/internal/autobahn/data -count=1
  • go test ./sei-cosmos/storev2/rootmulti -count=1
  • focused FlatKV GC tests: go test ./sei-db/state_db/sc/flatkv -run 'TestGC|TestGCLatestBlock|TestGCConcurrentWithCommitter' -count=1 -timeout=15m
  • go vet on the changed storage packages and sei-tendermint/autobahn/types
  • gofmt and git diff --check

A full FlatKV package run was attempted but exceeded its 10-minute package timeout in pre-existing snapshot/concurrency tests; the focused FlatKV GC suite passed.

@cursor

cursor Bot commented Aug 12, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Touches crash durability and retention logic across BlockDB, ReceiptDB, and StateWAL; a bug could over-prune recoverable history. Changes are deliberately conservative (retain extra data on failure) and the shared collector is not yet production-wired.

Overview
Makes storage pruning crash-consistent by measuring GC rollback windows and prune caps from each store's highest crash-recoverable block, not the newest buffered/written suffix.

BlockDB and StateWAL now track a durable tip advanced on Flush/Close (lastDurableBlockNumber / lastDurableBlock). GetLatestBlock and prune caps use that tip, so an unflushed suffix can no longer justify reclaiming durable history.

ReceiptDB persists a recovered m:latest_durable marker after Litt flushes, without promoting an index-only suffix left by an earlier crash. Tag-index range deletes and the m:earliest floor now commit in one synced Pebble batch (via batch DeleteRange) before the in-memory floor is published.

Contracts and tests are updated for pre-flush heads, durable prune caps, restart recovery, and receipt-floor commit ordering.

Reviewed by Cursor Bugbot for commit b81206d. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b81206d. Configure here.

if err := batch.Set(receiptEarliestVersionKey, encodeBlockNumber(cutoff)); err != nil {
return err
}
if err := s.index.Set(receiptEarliestVersionKey, encodeBlockNumber(cutoff), dbtypes.WriteOptions{}); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Local pruner uses volatile head

Medium Severity

startPruning still computes its keepRecent cutoff from latestVersion, while pruneBlocksBelow now caps at latestDurableVersion. When the write tip leads the durable tip, that cutoff can land at or above the durable head, so the floor advances too far and drops crash-recoverable receipts that still fall inside the configured retention window. Flush lag makes this an off-by-lag undercount; a stuck flusher can collapse retention to a single durable block.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit b81206d. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant