HDDS-9377. Replace MutableStat with lock-free ConcurrentMutableStat in OMLockMetrics and PerformanceMetrics

[yandrey321]

What changes were proposed in this pull request?

Problem

MutableStat.add(long) is synchronized. OMLockMetrics records four timing stats (read/write lock held and wait times) on every lock acquire and release. Under high read-lock concurrency — when many OM handler threads release a read-lock simultaneously and all rush to call add() — every thread serialises through the same mutex. This creates a thundering-herd contention point that limits OM throughput as thread counts grow.

Approach

Introduce ConcurrentMutableStat, a drop-in subclass of MutableStat that makes add() non-blocking:

Each call accumulates in LongAdder (sum, count) and LongAccumulator (min, max) — all cell-striped via Striped64, so threads write to independent cells with no cross-thread synchronisation.
setChanged() is deferred out of the hot-path add() to avoid concurrent volatile writes from all calling threads.
Pending cells are drained into the parent's running state lazily, only on snapshot(), lastStat(), or toString() — paths that already hold or take the stat's own lock and are called far less frequently than add().
OMLockMetrics and PerformanceMetrics are updated to use ConcurrentMutableStat for their stat fields.

Co-authored with Claude Opus 4.8

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9377

How was this patch tested?

Unit tests (TestConcurrentMutableStat): single-threaded count/mean/min/max correctness, multi-threaded count and extreme-value accuracy, multiple drain cycles.

Integration tests: TestFreon (OmBucketReadWriteFileOps, OmBucketReadWriteKeyOps) — both pass; verifyOMLockMetrics asserts sample counts > 0 for all four lock stats through the full cluster lock/unlock path.

Benchmark
x86

Threads    MutableStat µs/burst      ConcurrentMutableStat µs/burst    Speedup
--------------------------------------------------------------------------------------------
1          29.39 µs                  27.46 µs                          1.1x
10         744.88 µs                 150.38 µs                         5.0x
20         1.51 ms                   267.03 µs                         5.7x
40         2.91 ms                   518.87 µs                         5.6x
60         8.85 ms                   768.37 µs                         11.5x
80         7.40 ms                   996.14 µs                         7.4x

=== Steady-State (continuous load, 500 ms window) ===
Threads    MutableStat ops/ms        ConcurrentMutableStat ops/ms      Speedup
--------------------------------------------------------------------------------------------
1          35.83 k ops/ms            60.30 k ops/ms                    1.7x
10         4.40 k ops/ms             538.24 k ops/ms                   122.4x
20         4.85 k ops/ms             1.06 G ops/ms                     218.4x
40         4.71 k ops/ms             1.45 G ops/ms                     308.5x
60         5.42 k ops/ms             2.32 G ops/ms                     427.7x
80         5.73 k ops/ms             2.58 G ops/ms                     450.9x

ARM64

Threads    MutableStat µs/burst      ConcurrentMutableStat µs/burst    Speedup
--------------------------------------------------------------------------------------------
1          15.16 µs                  14.50 µs                          1.0x
10         177.60 µs                 235.61 µs                         0.8x
20         167.04 µs                 378.45 µs                         0.4x
40         336.60 µs                 1.28 ms                           0.3x
60         517.25 µs                 1.95 ms                           0.3x
80         791.09 µs                 2.40 ms                           0.3x

=== Steady-State (continuous load, 500 ms window) ===
Threads    MutableStat ops/ms        ConcurrentMutableStat ops/ms      Speedup
--------------------------------------------------------------------------------------------
1          292.66 k ops/ms           190.49 k ops/ms                   0.7x
10         10.77 k ops/ms            17.97 k ops/ms                    1.7x
20         9.96 k ops/ms             18.10 k ops/ms                    1.8x
40         10.22 k ops/ms            16.72 k ops/ms                    1.6x
60         12.40 k ops/ms            16.57 k ops/ms                    1.3x
80         9.44 k ops/ms             18.16 k ops/ms                    1.9x