[Subscription] Protect retained WAL by committed progress - #18399
Open
Caideyipi wants to merge 1 commit into
Open
[Subscription] Protect retained WAL by committed progress#18399Caideyipi wants to merge 1 commit into
Caideyipi wants to merge 1 commit into
Conversation
jt2594838
reviewed
Aug 6, 2026
Comment on lines
+3203
to
+3219
| final WALNode walNode = (WALNode) consensusReqReader; | ||
| final File[] walFiles = WALFileUtils.listAllWALFiles(walNode.getLogDirectory()); | ||
| if (Objects.isNull(walFiles) || walFiles.length == 0) { | ||
| return Math.max(0L, currentWalVersion); | ||
| } | ||
|
|
||
| WALFileUtils.ascSortByVersionId(walFiles); | ||
| for (final File walFile : walFiles) { | ||
| final long versionId = WALFileUtils.parseVersionId(walFile.getName()); | ||
| if (versionId >= currentWalVersion) { | ||
| return Math.max(0L, currentWalVersion); | ||
| } | ||
| if (ProgressWALIterator.isHeaderOnlyWalFile(walFile)) { | ||
| continue; | ||
| } | ||
|
|
||
| WalFileCommitRequirement requirement = walFileCommitRequirements.get(versionId); |
Contributor
There was a problem hiding this comment.
Is it possible to record the last visited WalFileCommitRequirement, and avoid listing files when it cannot still be covered by the committed progress.
The overhead of computing the id after each commit concerns me.
Comment on lines
+3376
to
+3384
| private static int compareProgress( | ||
| final WriterProgress leftProgress, final WriterProgress rightProgress) { | ||
| final int physicalTimeComparison = | ||
| Long.compare(leftProgress.getPhysicalTime(), rightProgress.getPhysicalTime()); | ||
| return physicalTimeComparison != 0 | ||
| ? physicalTimeComparison | ||
| : Long.compare(leftProgress.getLocalSeq(), rightProgress.getLocalSeq()); | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
WriterProgress does not implement compareTo?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Retain WAL required by uncommitted consumers
Previously, subscription WAL reclamation only considered replication progress and Topic retention policies. Consumer-group committed progress was not part of the deletion boundary, so retention could reclaim WAL still required by a lagging consumer.
This PR derives a file-level retained WAL version for each
(consumerGroup, topic, region)queue from its per-writer committedRegionProgress. IoTConsensus combines these boundaries with Topic size/time retention and uses the most conservative boundary.Multi-writer and leader migration handling
Committed progress remains represented by
(physicalTime, writerNodeId, localSeq)rather than localsearchIndex. Rolled WAL V3 metadata is summarized per writer, allowing follower WAL records and records produced before leader migration to be protected correctly.WAL files with incomplete or unsupported writer metadata are conservatively retained.
Boundary refresh and caching
The committed WAL boundary is refreshed after ACK, direct commit, initialization, seek, committed-progress broadcast, or WAL roll. Per-file writer-progress requirements are cached so repeated safe-delete checks do not repeatedly scan unchanged WAL metadata.
This PR has:
Tests
mvn spotless:apply -pl iotdb-core/consensus,iotdb-core/datanodemvn test -pl iotdb-core/consensus -Dtest=SubscriptionWalRetentionCalculatorTestmvn test -pl iotdb-core/consensus,iotdb-core/datanode -Dtest=ConsensusPrefetchingQueueTest#testWalFileCommitRequirementUsesPerWriterMaximum+testWalFileCommitRequirementRejectsUnsupportedWriterMetadata -Dsurefire.failIfNoSpecifiedTests=falsemvn test-compile -pl iotdb-core/consensus,iotdb-core/datanode -P with-zh-locale -DskipTestsKey changed/added classes
SubscriptionWalRetentionCalculatorSubscriptionQueueRegistryIoTConsensusServerImplConsensusPrefetchingQueue