Skip to content

HDDS-16133. Make TestReconTasks EMPTY_MISSING check deterministic - #11059

Open
Eason09053360 wants to merge 2 commits into
apache:masterfrom
Eason09053360:HDDS-16133
Open

HDDS-16133. Make TestReconTasks EMPTY_MISSING check deterministic#11059
Eason09053360 wants to merge 2 commits into
apache:masterfrom
Eason09053360:HDDS-16133

Conversation

@Eason09053360

@Eason09053360 Eason09053360 commented Aug 19, 2026

Copy link
Copy Markdown

What changes were proposed in this pull request?

TestReconTasks#testContainerHealthTaskDetectsEmptyMissingWhenAllReplicasLost times out
intermittently in CI. Raising the timeout does not help: the logs show
Stored 1 MISSING, 0 EMPTY_MISSING for the whole 20s wait, so the container is classified into
the wrong state rather than classified late.

Recon picks EMPTY_MISSING over MISSING in ReconReplicationManager#isEmptyMissing, i.e.
getNumberOfKeys() == 0. The test set up its container with runTestOzoneContainerViaDataNode(),
which writes a block, and that block count reaches Recon as a key count: putBlock increments
the datanode block count, the datanode reports it as the replica key count
(ContainerData#setContainerReplicaProto -> setKeyCount(blockCount)), and the report handler
copies it into the container metadata (AbstractContainerReportHandler#updateContainerUsedAndKeys).
getNumberOfKeys() is then 1, so the replica-less container is recorded as MISSING.

The race: handleCreateContainer sends an ICR before any block is written, so the test's
replica-sync wait is already satisfied by a report carrying keyCount=0, and the datanode is
shut down right afterwards. handlePutBlock sends no ICR, so keyCount=1 can only arrive with
the next periodic full container report (1s in this test). Whether it lands before the shutdown
is purely a matter of timing, which is why the test only fails under CI load.

The fix creates the container without writing a block, so the datanode always reports a key
count of 0 regardless of report timing. The javadoc, which claimed the key count stays 0 merely
because Ozone Manager is bypassed, is corrected as well.

HDDS-16133.001.patch on the JIRA proposes the same approach; this change was arrived at
independently.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16133

How was this patch tested?

The CI failure does not reproduce locally, since the race is always won on an unloaded machine.
The cause was confirmed by instrumenting the test instead: the registered replica consistently
showed keyCount=0, and both Recon's and SCM's numberOfKeys stayed 0 throughout. The local
runs were already hitting the post-fix state by accident; this change makes it guaranteed.

Generated-by: Claude Code (Claude Opus 5)

testContainerHealthTaskDetectsEmptyMissingWhenAllReplicasLost wrote a block
into the container via runTestOzoneContainerViaDataNode before killing the
datanode. putBlock increments the datanode's block count, which container
reports propagate into ContainerInfo#numberOfKeys. Recon classifies
EMPTY_MISSING via getNumberOfKeys() == 0, so once a report carrying a
non-zero key count landed, the container was recorded as MISSING and the
20s await could never succeed.

The container-create ICR is sent before the block is written, so the test's
replica-sync wait is already satisfied by a report carrying keyCount=0.
Whether the next 1s full container report (keyCount=1) arrives before the
datanode is stopped is purely a timing race, which is why this only fails
under CI load.

Create the container without writing a block so the datanode always reports
a key count of 0, and correct the javadoc that claimed the key count stays 0
merely because Ozone Manager is bypassed.

Generated-by: Claude Code (Claude Opus 5)
Copilot AI lite review requested due to automatic review settings August 19, 2026 06:25
@Eason09053360
Eason09053360 marked this pull request as ready for review August 19, 2026 06:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Makes the Recon integration test for EMPTY_MISSING classification deterministic by ensuring the test container is created without writing any block data, avoiding a timing-dependent key-count propagation race between datanode reports and Recon’s container metadata.

Changes:

  • Update testContainerHealthTaskDetectsEmptyMissingWhenAllReplicasLost to create the container via an explicit CreateContainer command (no block write), keeping numberOfKeys == 0 reliably.
  • Correct the test javadoc to reflect the actual reason the key count stays 0 (empty container / replica key count) rather than “bypassing OM” alone.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@chihsuan chihsuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch! @Eason09053360 LGTM +1

I left one optional nit about simplifying the Javadoc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants