Skip to content

25.3 Stable Backport of #84317 - Report cgroup and system-wide metrics along with each other - #2333

Merged
mkmkme merged 2 commits into
releases/25.3.14from
backports/25.3.14/84317
Sep 11, 2026
Merged

25.3 Stable Backport of #84317 - Report cgroup and system-wide metrics along with each other#2333
mkmkme merged 2 commits into
releases/25.3.14from
backports/25.3.14/84317

Conversation

@mkmkme

@mkmkme mkmkme commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Report cgroup and system-wide metrics along with each other

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Cgroup-level and system-wide metrics are reported now altogether. Cgroup-level metrics have names CGroup<Metric> and OS-level metrics (collected from procfs) have names OS<Metric> (ClickHouse#84317 by @nickitat).

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

Report cgroup and system-wide metrics along with each other
@mkmkme

mkmkme commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

@blau-ai

@blau-ai

blau-ai commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

CI triage

Verdict: 0 PR-caused failures. All red checks are pre-existing flaky integration tests, an unrelated regression-suite assertion, a DCO/sign-off check, and the aggregate gate that just reflects them. Notably, this PR's own modified test — test_async_metrics_in_cgroup — passes in CI, which is the direct signal that the backported AsynchronousMetrics change behaves as intended.

The diff only touches src/Common/AsynchronousMetrics.{cpp,h} (new CGroup*Time[Normalized] async metrics) and tests/integration/test_async_metrics_in_cgroup/test.py. None of the failing tests exercise that code path.

Per-failure breakdown

1. Integration tests (asan) [5/8] and (release) [5/8] (= job runners IntegrationTestsAsan-4, IntegrationTestsRelease-4, IntegrationTestsAnalyzerAsan-4)

  • Single failure in each: test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race (fail: 1, passed: 427/429).
  • Classification: flaky, not PR-related. It's a deliberate ReplicatedMergeTree partition move/drop race test with zero connection to async metrics. [5/8] is the 1-indexed report for the 0-indexed -4 shard, which is why those three job rows all point at the same test.
  • Next step: safe to re-run.

2. IntegrationTestsAnalyzerAsan-0 (old analyzer, shard [1/8])

  • Failures (from the job log): test_async_insert_adaptive_busy_timeout::test_change_queries_frequency, test_cluster_discovery::test_cluster_discovery_startup_and_stop, test_parallel_replicas_custom_key_load_balancing[...], test_parallel_replicas_over_distributed[...].
  • Classification: flaky/infra, not PR-related. All timing- and replica-coordination-sensitive tests that are known to be unstable on loaded CI runners; none touch metrics. (The regular-analyzer asan [1/8] shard passed, consistent with flakiness.)
  • Next step: re-run.

3. RegressionTestsRelease / Common (version) / version (a.k.a. Regression release version)

  • Failing scenario: no_clickhouse_cloud_url_in_auth_error in version/tests/altinity_version.py. It asserts the wrong-password auth error does not contain https://clickhouse.cloud, but this build's AUTHENTICATION_FAILED message still includes:

    If you use ClickHouse Cloud, the password can be reset at https://clickhouse.cloud/ ...

  • Classification: not PR-related (pre-existing build/branding gap). This is about the authentication error text, an entirely different subsystem from AsynchronousMetrics. The regression suite (release branch) checks for an Altinity patch that strips the Cloud URL from the auth message; that patch isn't present in this 25.3.14 build. It will fail identically on releases/25.3.14 regardless of this PR.
  • Next step: belongs to the release/branding backlog, not this backport — compare against a base-branch (releases/25.3.14) regression run to confirm it's independent.

4. DCO

5. FinishCheck

  • Aggregate gate; red only because the checks above are red. No independent failure. Goes green once the flaky suites are re-run (and DCO/regression are waived).

Bottom line

Nothing here indicates a problem introduced by the backport. Re-running the two integration suites should clear items 1–2; items 3–4 are pre-existing/process issues to be waived or handled outside this PR. No code changes proposed.

@blau-ai (automated CI triage). Evidence: praktika S3 reports for 188b1608 + job logs from run 34531380972.

@mkmkme mkmkme added the verified Approved for release label Sep 11, 2026
@mkmkme
mkmkme merged commit cecb04c into releases/25.3.14 Sep 11, 2026
397 of 411 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants