Skip to content

feat(control-plane): lower PromQL with ingestion cadence - #741

Merged
zzylol merged 4 commits into
mainfrom
fix/issue-729
Sep 18, 2026
Merged

zzylol merged 4 commits into
mainfrom
fix/issue-729

Conversation

@zzylol

@zzylol zzylol commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Closes #729

Uses merged ASAPPlanner #417 (pinned at merge commit c9b2aa78), including the merged peer-workload API from ASAPPlanner #442.

Before this PR

Instant selectors lost their source horizon during planning. Backend current-series validation assumed a five-minute horizon, physical alternatives reparsed queries with compatibility defaults, and HTTP planning bypassed workload-aware lowering.

After this PR

A top-level data_workload declaring a 1 s ingestion interval makes sum(data) lower as Aggregate → TimeRange(1 s) → Scan. Snapshot and HTTP planning pass query and data workloads as peers, check evidence at the planning clock, and preserve the original IR through physical alternatives. Current-series execution expires members at the declared horizon; explicit range selectors retain their query-owned ranges.

Changes

  • Update the Planner pin and lifecycle-demand calls for the peer-workload API.
  • Preserve workload-lowered roots during alternative enumeration and exact fallback; match residuals with horizon-aware, full-IR equality.
  • Accept positive runtime horizons and cap input lag at that horizon.
  • Migrate examples and discovery output to top-level data evidence, including observed cadence; retain explicit legacy scrape-interval migration.
  • Handle newly introduced Planner temporal types conservatively as unsupported in the ClickHouse adapter.
  • Test freshness rejection, one-second expiry, residual equality, discovery, and process-level remote-write ingestion/query/expiry. Uncertified quantile ratios retain exact execution.

Validation

Ready for review. Uses the merged upstream ASAPPlanner #417 revision; w14 remains fallback-only.

See the query results and reproducible E2E commands.

@zzylol
zzylol marked this pull request as ready for review September 18, 2026 17:53
@zzylol

zzylol commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

#702 compilation and non-fallback execution results

Tested against backend 661cc125 and the pinned ASAPPlanner #417 revision 1f1f0993 (including the peer-workload API from ASAPPlanner #442). All nine queries compile, but only eight have an accelerated execution path under these test inputs:

Query Executes through ASAP without fallback
quantile by (job) (0.9, data) Yes
sum by (job) (data) Yes
sum(data) Yes
avg_over_time(data[5m]) Yes
avg_over_time(data[5m]) / quantile_over_time(0.5, data[5m]) No: ExactFallback
count by (job) (data) Yes
count(data) Yes
avg by (job) (data) Yes
sum by (job) (sum_over_time(data[5m])) Yes

What was tested

Each query was tested independently with a 1-second source ingestion interval. Non-quantile queries request exact accuracy; quantile-containing queries use epsilon=0.01, delta=0.01. The fixture renames data to issue701_data.

The test invokes the pinned Planner, enumerates/compiles physical alternatives, and requires a plan without external-exact nodes. Synthetic cost quotes favor the fully accelerated alternative. It then starts the real backend executable, installs the snapshot, ingests Remote Write samples for three series across two jobs, and checks nonempty ASAP responses after two successive ingestion batches. The default mock Prometheus serves health only, not query results.

This demonstrates acceleration capability, not measured speedup, production cost-based selection, or a real-Prometheus numerical differential run.

Reproduce on this PR

Prerequisites: Rust, protoc, and Git access to the Cargo dependencies. No Docker or external Prometheus is needed. Run from the backend repository with a clean checkout:

gh pr checkout 741
unset ASAP_CURRENT_SERIES_PROMETHEUS_URL

# All eight supported #702 queries, independently, through the real process.
cargo test --locked -p data_plane \
  --test asapquery_compatibility_process_e2e \
  issue_702_individual_queries_execute_without_fallback \
  -- --nocapture --test-threads=1

# Assert the known ratio limitation instead of counting fallback as acceleration.
cargo test --locked -p data_plane \
  --test asapquery_compatibility_process_e2e \
  issue_701_702_uncertified_ratios_require_exact_fallback \
  -- --nocapture --test-threads=1

# Combined issue coverage, including mixed workloads and average overflow.
cargo test --locked -p data_plane \
  --test asapquery_compatibility_process_e2e issue_ \
  -- --nocapture --test-threads=1

If protoc is not on PATH, prefix Cargo commands with PROTOC=/absolute/path/to/protoc.

CI fix

The failing mixed-workload test incorrectly required uncertified quantile ratios to accelerate. The fix separates supported warm workloads from explicit ratio-fallback coverage, sets the peer data workload's cadence to 1 second, and adds the permanent per-query #702 process regression. The warm checks still reject exact fallback, external exact execution, and exact/candidate-exact subqueries, and require execution: asap in runtime responses.

The focused issue tests passed (4/4). Full workspace validation and the new GitHub CI run are being checked separately. This does not implement w14 acceleration.

@zzylol

zzylol commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Validation complete for a8f7572: the full local workspace suite passed (1,789 passed, 0 failed, 2 ignored), workspace Clippy passed with warnings denied, and GitHub CI is green. The permanent #702 per-query process test passes, as does the corrected mixed-workload test. w14 acceleration remains unsupported; its exact-fallback expectation is covered separately.

@zzylol
zzylol merged commit 44653cc into main Sep 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make PromQL lowering workload-aware and require data ingestion interval

1 participant