HDDS-15071. [SCM] Add configuration and global EC reconstruction limit by jojochuang · Pull Request #10122 · apache/ozone

jojochuang · 2026-04-24T17:06:04Z

What changes were proposed in this pull request?

HDDS-15071. [SCM] Add configuration and global reconstruction limit

Please describe your PR in detail:

Introduce the foundational configuration properties in ReplicationManagerConfiguration:

hdds.scm.replication.decommission.ec.reconstruction.enabled
hdds.scm.replication.decommission.ec.reconstruction.load.factor (default 0.9)
hdds.scm.replication.reconstruction.global.limit (default 0, disabled)

Implement an atomic counter in ReplicationManager to track active ReconstructECContainersCommand tasks cluster-wide. Enforce the global limit in sendThrottledReconstructionCommand() via tryReserveReconstructionSlot() so 1-1 replication continues when the reconstruction cap is reached.

Default behavior is unchanged. hdds.scm.replication.reconstruction.global.limit defaults to 0, which disables cluster-wide reconstruction limit checking (same pattern as hdds.scm.replication.inflight.limit.factor). Set a positive value to opt in to throttling; a value of 50 is recommended when enabling HDDS-15072 EC decommission reconstruction.

The inflight reconstruction counter is always maintained for observability, even when the limit is disabled. Limit enforcement only applies when reconstruction.global.limit > 0.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15071

How was this patch tested?

Unit tests:

TestReplicationManager.testReconstructionGlobalLimitDisabledByDefault()
TestReplicationManager.testInflightReconstructionLimit()
TestReplicationManager.testSendReconstructionCommandRejectedWhenGlobalLimitReached()
TestReplicationManager.testReconstructionGlobalLimitEnforcedConcurrently()
TestReplicationManager.testNotifyStatusChangedClearsReconstructionCounters()
TestReplicationManager.testInflightReconstructionCountNotNegativeAfterFailoverClear()
TestUnderReplicatedProcessor.testMessageNotProcessedIfGlobalLimitReached() (replication inflight limit)

smengcl

lgtm

jojochuang · 2026-07-06T22:32:05Z

Pushed a small change to disable the global reconstruction threshold, to keep the behavior the same.

smengcl

Thanks @jojochuang . I found two more issues

Copilot

Pull request overview

Adds a cluster-wide throttling mechanism for EC reconstruction in SCM’s ReplicationManager, backed by new configuration knobs. This helps prevent excessive aggregate EC reconstruction work when the feature is enabled, while keeping default behavior unchanged (limit disabled by default).

Changes:

Introduces new ReplicationManagerConfiguration properties for EC decommission reconstruction enablement, load factor, and a global EC reconstruction limit.
Tracks inflight EC reconstruction commands via an atomic counter and per-command fragment accounting, and enforces the global limit when sending reconstruction commands.
Adds/extends unit tests covering default-disabled behavior, limit accounting, and limit-triggered rejection paths.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java	Adds global EC reconstruction limit config + inflight tracking and enforcement in reconstruction command sending / completion handling.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestReplicationManager.java	Adds tests for default behavior, inflight reconstruction tracking, rejection at limit, and counter reset on leader transitions.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestUnderReplicatedProcessor.java	Adds a processor test related to reconstruction-limit behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java:601

The PR description says UnhealthyReplicationProcessor will check the global reconstruction limit before dequeuing work; however, the processors only check the replication inflight limit and there is no reconstruction-limit check in UnhealthyReplicationProcessor/UnderReplicatedProcessor. As implemented, the limit is enforced only when sending the command (by throwing CommandTargetOverloadedException), which can still dequeue/process/requeue items and create churn. Please either implement the dequeue-time check or update the PR description/testing notes accordingly.

    if (isReconstructionLimitReached()) {
      metrics.incrECReconstructionCmdsDeferredTotal();
      throw new CommandTargetOverloadedException(
          "Global reconstruction limit (" + getReconstructionInFlightLimit()
              + ") reached for container " + containerInfo.getContainerID());
    }
    List<DatanodeDetails> targets = command.getTargetDatanodes();
    List<Pair<Integer, DatanodeDetails>> targetWithCmds =
        getAvailableDatanodesForReplication(targets);
    if (targetWithCmds.isEmpty()) {
      metrics.incrECReconstructionCmdsDeferredTotal();
      throw new CommandTargetOverloadedException("No target with capacity " +
          "available for reconstruction of " + containerInfo.getContainerID());
    }
    DatanodeDetails target = selectAndOptionallyExcludeDatanode(
        rmConf.getReconstructionCommandWeight(), targetWithCmds);
    sendDatanodeCommand(command, containerInfo, target);
  }

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 9 comments.

Set hdds.scm.replication.reconstruction.global.limit default to 0 so cluster-wide EC reconstruction throttling is opt-in and default behavior matches pre-PR Ozone. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I6a0915cf8b181a1a98b08a4f3ede071e4d460e88

… reset. Move global reconstruction throttling from the under-replicated processor loop to sendThrottledReconstructionCommand so 1-1 replication continues when the cap is reached. Clear reconstruction counters on leader transition when pending ops are cleared. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I881fee0c3ea3c2e350b8fbb95321985569bef636

Skip reconstruction counter decrements when the command is no longer tracked after notifyStatusChanged clear, and floor decrements at zero. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I1530348aa2b09c232ad0410468d90cf098d8d0a3

…and tests. Validate reconstruction config bounds, remove a misleading processor test stub, and reuse ECUnderReplicationHandler.integers2ByteString in unit tests. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I55216c0c4c0af92e003486e04dd9bc0c42a8dc5c

…espace. Reserve the global reconstruction limit with a CAS before sending commands, release on send failure, and add a concurrent unit test. Clean up trailing whitespace in reconstruction limit tests. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: If8c0e4613209b8fb381271c7dab1733eac0e5bde

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

…ress review nits. Always reserve reconstruction slots via CAS even when the global limit is disabled so inflightReconstructionCount stays accurate for observability and runtime reconfiguration. Extend the default-disabled test, wrap long test lines, rename the concurrent-test command ID variable, and shut down the executor in a finally block. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I92d085a415582483d9baa3a8d42b1a956b42e485

jojochuang · 2026-07-16T22:23:55Z

@smengcl Thanks for the July 7 review. Both items are addressed in subsequent commits:

notifyStatusChanged counter reset — reconstructionCommandIdToPendingFragmentCount and inflightReconstructionCount are cleared when pending ops are cleared on leader transition (fd8f55ca459 / 61e4eac3650).
Processor break halting all under-replication — enforcement was moved from UnhealthyReplicationProcessor to sendThrottledReconstructionCommand() so 1-1 replication continues when the reconstruction cap is reached (61e4eac3650).

Latest push (aad8da6d69d) also:

always tracks inflight reconstruction count (even when limit is disabled) for observability and runtime reconfig
uses atomic CAS reservation before send (c196dd48a0f)
updates the PR description to match send-path enforcement

All review threads are resolved. Could you take another look when you have a chance?

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

…g ops. Register reconstructionCommandIdToPendingFragmentCount immediately after reserving a slot so opCompleted can always release it. Remove the map put from adjustPendingOpsAndMetrics and clear the entry on send failure. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I454d0a6d8510a200d93a26118cda8990416d149f

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

github-actions · 2026-08-09T00:08:04Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

github-actions · 2026-08-17T00:03:53Z

Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.

jojochuang · 2026-08-18T21:08:29Z

Rebased onto latest master and force-pushed. Unit tests for global reconstruction limit pass locally:

mvn -pl :hdds-server-scm -am test -Dtest=TestReplicationManager#testReconstructionGlobalLimit* -DskipShade -DskipRecon -DskipDocs

@smengcl Could you take another look when you have a chance? This is the foundation for HDDS-15072 (dynamic EC decommission reconstruction switch).

jojochuang · 2026-08-18T21:12:58Z

Addressed @adoroszlai review in 6aa5640 (also on #11054):

Removed @VisibleForTesting from getReconstructionPendingFragmentCount
Changed return type to int using getOrDefault(cmdId, 0)
Updated tests accordingly

Review threads resolved.

adoroszlai reviewed Apr 24, 2026

View reviewed changes

Comment thread hadoop-ozone/common/pom.xml Outdated

jojochuang changed the title ~~Hdds 15071~~ HDDS-15014. [SCM] Add configuration and global reconstruction limit Apr 24, 2026

jojochuang changed the title ~~HDDS-15014. [SCM] Add configuration and global reconstruction limit~~ HDDS-15071. [SCM] Add configuration and global reconstruction limit Apr 24, 2026

jojochuang force-pushed the HDDS-15071 branch from daf857e to d4fdcbe Compare April 24, 2026 17:59

adoroszlai mentioned this pull request May 11, 2026

WIP. HDDS-15014. Speed up EC container decommission #10082

Closed

jojochuang requested a review from smengcl May 27, 2026 14:06

smengcl approved these changes May 27, 2026

View reviewed changes

jojochuang marked this pull request as ready for review June 18, 2026 19:26

smengcl reviewed Jul 7, 2026

View reviewed changes

Comment thread ...r-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java

Comment thread ...ain/java/org/apache/hadoop/hdds/scm/container/replication/UnhealthyReplicationProcessor.java Outdated

jojochuang requested a review from Copilot July 9, 2026 00:05

Copilot started reviewing on behalf of jojochuang July 9, 2026 00:05 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

jojochuang requested a review from Copilot July 9, 2026 01:14

Copilot started reviewing on behalf of jojochuang July 9, 2026 01:15 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

jojochuang requested a review from Copilot July 9, 2026 01:28

Copilot started reviewing on behalf of jojochuang July 9, 2026 01:29 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

jojochuang changed the title ~~HDDS-15071. [SCM] Add configuration and global reconstruction limit~~ HDDS-15071. [SCM] Add configuration and global EC reconstruction limit Jul 9, 2026

jojochuang and others added 6 commits July 10, 2026 10:14

HDDS-15071. [SCM] Add configuration and global reconstruction limit

ed4e47c

Copilot AI review requested due to automatic review settings July 10, 2026 17:14

jojochuang force-pushed the HDDS-15071 branch from b5ca933 to c196dd4 Compare July 10, 2026 17:14

Copilot started reviewing on behalf of jojochuang July 10, 2026 17:14 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread ...r-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java Outdated

Comment thread ...r-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java

Copilot AI review requested due to automatic review settings July 16, 2026 22:23

Copilot started reviewing on behalf of jojochuang July 16, 2026 22:23 View session

Copilot AI reviewed Jul 16, 2026

jojochuang requested a review from Copilot July 16, 2026 22:29

Copilot started reviewing on behalf of jojochuang July 16, 2026 22:29 View session

Copilot AI reviewed Jul 16, 2026

Copilot AI review requested due to automatic review settings July 16, 2026 22:57

Copilot started reviewing on behalf of jojochuang July 16, 2026 22:58 View session

Copilot AI reviewed Jul 16, 2026

adoroszlai requested a review from smengcl July 18, 2026 19:14

adoroszlai reviewed Jul 18, 2026

View reviewed changes

Comment thread ...r-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java

Comment thread ...r-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManager.java

github-actions Bot added the stale label Aug 9, 2026

github-actions Bot closed this Aug 17, 2026

jojochuang mentioned this pull request Aug 18, 2026

HDDS-15071. [SCM] Add configuration and global EC reconstruction limit #11054

Open

3 tasks

Uh oh!

Conversation

jojochuang commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Uh oh!

smengcl left a comment

Choose a reason for hiding this comment

Uh oh!

jojochuang commented Jul 6, 2026

Uh oh!

smengcl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

jojochuang commented Jul 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 9, 2026

Uh oh!

github-actions Bot commented Aug 17, 2026

Uh oh!

jojochuang commented Aug 18, 2026

Uh oh!

jojochuang commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

jojochuang commented Apr 24, 2026 •

edited

Loading