Skip to content

fix(clickhouse): reuse installed SQL plans for moving time windows - #691

Merged
zzylol merged 3 commits into
mainfrom
fix/sql-moving-window-binding
Sep 12, 2026
Merged

zzylol merged 3 commits into
mainfrom
fix/sql-moving-window-binding

Conversation

@zzylol

@zzylol zzylol commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Why

Dashboard refreshes change SQL timestamp bounds, so fixed canonical identities miss an installed plan even when its summaries cover the refreshed window.

What

Reuse same-duration time windows for simple single-table aggregates (Project? → Aggregate → Scan). Preserve thresholds, projection constants and other query semantics. Multiple concrete windows can coexist without losing their individual materializations or pane origins. Keep old fixed publications compatible and external SQL subqueries fixed.

How

Normalize resolved integer source bounds into a half-open window template. Keep concrete SQL identities as primary catalog keys and add a separate optional ClickHousePlanningContext.window_templates index referencing those entries. Validate template references at publication. Serving first checks concrete identities, then tries indexed entries with the request's actual range until one passes execution and coverage checks.

Support >=, >, <, <= and checked constant integer arithmetic. Missing template metadata defaults to an empty index, preserving old publications. New template-index publications require the updated backend.

Before this PR

An installed [0,1000) aggregate cannot match a refresh for [1000,2000) because the timestamp literals change its catalog key.

After this PR

The refresh reuses the installed computation and reads the new panes. The execution regression seeds consecutive panes with sums 2, 3 and 5, retains two concrete query entries, and verifies that the original windows and a later refresh each read their own data (20, 30 and 50 after the fixture's projection). Both compilation paths accept multiple concrete windows sharing a template; automatic compilation retains different materialization bindings for different pane origins. Missing or partial coverage falls back; a changed duration does not reuse the template.

Verification

  • Confirmed the original moving-window identity and inclusive/arithmetic-bound regressions failed before their fixes.
  • Confirmed the multi-window publication regression failed with duplicate-template rejection before separating the index.
  • Control-plane ClickHouse tests: 6 passed, covering both publication paths, distinct physical bindings and invalid template references.
  • Query-plan tests: 5 passed, including language identity and serialization checks.
  • Data-plane accelerator tests: all 10 passed with CLICKHOUSE_URL configured, including real ClickHouse backfill and mixed-execution differential tests.
  • Real ClickHouse process integration suite: all 4 tests passed (clickhouse_differential_e2e, --test-threads=1). The new test automatically publishes two concrete windows, installs/activates the plan in a backend process, backfills both materializations from ClickHouse, and compares HTTP query results with the native oracle. Original windows return 5 and 12, a new window returns 24, equivalent inclusive/arithmetic bounds return 24, all with warm routing. A window with raw data but no summary returns 36 through exact_fallback.
  • Live tests used an isolated ClickHouse 26.8.2.7 container, image ID sha256:fa394da808cc53f76d0344429421d6c422a6ee85fe7450135c0e3cff4df9bcbb, 2 CPUs / 2 GiB, with authentication enabled. The test container was removed afterward.
  • Rustfmt on changed files and git diff --check passed.
  • Runtime tests used an isolated source copy with compatible sketchlib revision 8c03d7c, because the local shared checkout lacks methods required by main. The shared dependency checkout was not changed.
  • Visual evidence: not applicable.

Limitations

Complex query shapes retain fixed matching. Refreshes still require complete, aligned summary panes; this change does not alter pane granularity, retention, or maintenance scheduling. This PR establishes query-plan reuse, not measured end-to-end performance benefits.

@zzylol
zzylol merged commit 3288aa1 into main Sep 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant