HDDS-16133. Make TestReconTasks EMPTY_MISSING check deterministic - #11059
Open
Eason09053360 wants to merge 2 commits into
Open
HDDS-16133. Make TestReconTasks EMPTY_MISSING check deterministic#11059Eason09053360 wants to merge 2 commits into
Eason09053360 wants to merge 2 commits into
Conversation
testContainerHealthTaskDetectsEmptyMissingWhenAllReplicasLost wrote a block into the container via runTestOzoneContainerViaDataNode before killing the datanode. putBlock increments the datanode's block count, which container reports propagate into ContainerInfo#numberOfKeys. Recon classifies EMPTY_MISSING via getNumberOfKeys() == 0, so once a report carrying a non-zero key count landed, the container was recorded as MISSING and the 20s await could never succeed. The container-create ICR is sent before the block is written, so the test's replica-sync wait is already satisfied by a report carrying keyCount=0. Whether the next 1s full container report (keyCount=1) arrives before the datanode is stopped is purely a timing race, which is why this only fails under CI load. Create the container without writing a block so the datanode always reports a key count of 0, and correct the javadoc that claimed the key count stays 0 merely because Ozone Manager is bypassed. Generated-by: Claude Code (Claude Opus 5)
Contributor
There was a problem hiding this comment.
Pull request overview
Makes the Recon integration test for EMPTY_MISSING classification deterministic by ensuring the test container is created without writing any block data, avoiding a timing-dependent key-count propagation race between datanode reports and Recon’s container metadata.
Changes:
- Update
testContainerHealthTaskDetectsEmptyMissingWhenAllReplicasLostto create the container via an explicitCreateContainercommand (no block write), keepingnumberOfKeys == 0reliably. - Correct the test javadoc to reflect the actual reason the key count stays 0 (empty container / replica key count) rather than “bypassing OM” alone.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
chihsuan
approved these changes
Aug 21, 2026
chihsuan
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the patch! @Eason09053360 LGTM +1
I left one optional nit about simplifying the Javadoc.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
TestReconTasks#testContainerHealthTaskDetectsEmptyMissingWhenAllReplicasLosttimes outintermittently in CI. Raising the timeout does not help: the logs show
Stored 1 MISSING, 0 EMPTY_MISSINGfor the whole 20s wait, so the container is classified intothe wrong state rather than classified late.
Recon picks
EMPTY_MISSINGoverMISSINGinReconReplicationManager#isEmptyMissing, i.e.getNumberOfKeys() == 0. The test set up its container withrunTestOzoneContainerViaDataNode(),which writes a block, and that block count reaches Recon as a key count:
putBlockincrementsthe datanode block count, the datanode reports it as the replica key count
(
ContainerData#setContainerReplicaProto->setKeyCount(blockCount)), and the report handlercopies it into the container metadata (
AbstractContainerReportHandler#updateContainerUsedAndKeys).getNumberOfKeys()is then 1, so the replica-less container is recorded asMISSING.The race:
handleCreateContainersends an ICR before any block is written, so the test'sreplica-sync wait is already satisfied by a report carrying
keyCount=0, and the datanode isshut down right afterwards.
handlePutBlocksends no ICR, sokeyCount=1can only arrive withthe next periodic full container report (1s in this test). Whether it lands before the shutdown
is purely a matter of timing, which is why the test only fails under CI load.
The fix creates the container without writing a block, so the datanode always reports a key
count of 0 regardless of report timing. The javadoc, which claimed the key count stays 0 merely
because Ozone Manager is bypassed, is corrected as well.
HDDS-16133.001.patchon the JIRA proposes the same approach; this change was arrived atindependently.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16133
How was this patch tested?
flaky-test-check10x10 (100 runs), all splits green:https://github.com/Eason09053360/ozone/actions/runs/32270542328
mvn -pl :ozone-integration-test-recon test -Dtest=TestReconTasks— all 6 tests pass../hadoop-ozone/dev-support/checks/checkstyle.sh— 0 failures.The CI failure does not reproduce locally, since the race is always won on an unloaded machine.
The cause was confirmed by instrumenting the test instead: the registered replica consistently
showed
keyCount=0, and both Recon's and SCM'snumberOfKeysstayed 0 throughout. The localruns were already hitting the post-fix state by accident; this change makes it guaranteed.
Generated-by: Claude Code (Claude Opus 5)