docs: define backend plan split and SDS state contract by zzylol · Pull Request #737 · ProjectASAP/ASAPQuery-backend

zzylol · 2026-09-17T18:23:23Z

Define how one selected ASAPPlanner semantic DAG becomes separate backend-executable PrecomputePlan and QueryPlan subgraphs connected through SDS state references.

Design:

PrecomputePlan owns maintenance inputs, summary construction and stored-state writes.
QueryPlan owns bound state reads, SummaryEstimate, exact residuals and result composition.
Compiler bindings produce both plans and their catalog entries from one decision, preventing schema and identity drift.
Materialization, MaintenanceInput, Query and QueryInput describe semantic-to-physical mappings without placing query-only nodes in PrecomputePlan.
The plans and catalog install as one backend generation; readiness remains separate from activation.

SDS contract:

Defines SummaryDefinition, Materialization, SummaryStateInstance and StateReference as distinct objects.
Specifies identity, schema, generation, readiness, lifecycle and derived-state invariants.
Keeps payloads in the summary store and requires exact bound lookup rather than serving-time catalog search.

Migration scope:

Extract runtime-independent contracts and sketch reconstruction into neutral libraries.
Remove backend build/runtime/test dependencies on ASAPCollector.
Version and normalize supported legacy artifacts before retiring complete-DAG precompute execution.
CollectorPlan, TransmissionPlan, distributed activation and new transport behavior are deferred.

Verification: git diff --check passes. Local documentation links and anchors, including inbound references from docs/, pass. Documentation only; the plan split and library extraction are proposed implementation work.

This reverts commit daa5281.

milindsrivastava1997 · 2026-09-18T13:26:26Z

+## Purpose and scope

-## Design decision
+This design splits one selected ASAPPlanner semantic DAG into two executable


What is a "semantic DAG"? Is this the output of ASAPPLanner? How is this different from SummaryMaintenanceLifecyclePlan?

“semantic DAG” means the selected post-ASAP computation DAG produced by ASAPPlanner.
SummaryMaintenanceLifecyclePlan means after a post-ASAP DAG being generated, some logic of summary maintainance will output the plan for how a summary is maintiend, incremental vs built from data at rest. So these are two steps currently in the code.

I didn't understand the relationship between Semantic DAG and SummaryMaintenanceLifecyclePlan. Also, what is ASAPQuery-backend inputting from ASAPPlanner? One of these or both?

this relationship will be answered in ProjectASAP/ASAPPlanner#445

The backend uses both, at different stages:

Selected post-ASAP DAG: The backend’s selection adapter obtains a Rc root and passes it to physical compilation as QueryCompilationInput.selected_plan_root. This describes the
selected computation. Code

Maintenance lifecycle decisions: During physical compilation, select_lifecycle() calls Planner’s plan_summary_maintenance_lifecycles() with the selected summary node, workload demand,
capabilities, and cost evidence. That returns a SummaryMaintenanceLifecyclePlan. The backend extracts its selected lifecycle guarantee, window framework, implementation identity, and
costing information into backend configuration. Code

So the current flow is:

Select post-ASAP computation
→ pass selected root to backend compiler
→ compiler calls Planner for summary lifecycle decisions
→ combine computation and selected maintenance decisions
→ generate backend plans

The lifecycle plan refers to the summary computation it is planning maintenance for. It does not replace the DAG, and the backend does not simply execute the lifecycle-plan object directly.

Following up on my explanation above: that describes the current backend call sequence. Based on ASAPPlanner #445, I propose moving lifecycle-aware selection before physical compilation and passing its result directly to the compiler.

Current implementation

Select complete Post-ASAP DAG -> backend physical compiler receives selected_plan_root -> extracts summary producers -> calls Planner lifecycle API for selected producers -> extracts lifecycle/window decisions -> combines those decisions with the complete query DAG -> generates backend physical plans

The current compiler calls select_lifecycle(..., &selected.node, ...) on extracted producers (code). Those producer-local lifecycle results do not necessarily contain downstream query readouts, so the compiler still needs the separate complete query root.

Proposed integration

PlanningWorkload + evidence + capabilities -> ASAPPlanner -> PlanSpace -> lifecycle-aware selection and materialization -> SummaryMaintenanceLifecyclePlan - complete selected Post-ASAP root - maintenance decisions for its summary producers - summary-versus-raw recomputation decision -> backend physical compiler validates and binds the selected result -> PrecomputePlan -> QueryPlan -> Summary Catalog definitions -> installation and execution

Boundary Current implementation Proposed integration

Computation supplied to compiler Complete selected DAG root Complete root retained in the lifecycle-aware result

Lifecycle selection Called from inside physical compilation for extracted producers Completed through Planner's helper before physical compilation

Connecting DAG and maintenance decisions Backend combines the separate call results Compiler consumes their association in the selected result

Compiler responsibility Obtains lifecycle decisions and generates physical plans Validates feasibility and generates physical plans from the selected decisions

Raw recomputation Must be handled by the existing selection/compilation paths Explicitly honor the helper's raw-versus-summary decision

PlanSpace remains Planner's canonical logical output; lifecycle-aware selection is a helper over that output, as described in #445. The compiler still needs physical implementation, schema, storage and installation context. The simplification concerns the computation/lifecycle handoff, not removal of backend responsibilities.

Relative to the current PR #737 design, this is a smaller change: the document already requires the selected DAG plus lifecycle commitments. The proposal makes their handoff one associated result instead of independently supplied artifacts. The PrecomputePlan/QueryPlan split, catalog definitions and state references remain applicable.

The essential condition is that the result retains the complete query root, including readouts and remaining query operations. For multiple queries, preserve query-to-root associations and shared producer identity across the results so compilation does not duplicate maintenance. If raw recomputation is selected, do not unconditionally install summary producers.

This is a target integration proposal, not an interface the current backend already implements. It requires moving the lifecycle-selection call boundary and preserving those associations, not merely changing a parameter type.

milindsrivastava1997 · 2026-09-18T13:31:40Z

+
+## Architecture at a glance
+
+The current `PrecomputePlan.executable_dags` can contain a complete semantic DAG,


Pls add a note that this is confusing and must be changed. Related #740

milindsrivastava1997 · 2026-09-18T13:32:47Z

+lifecycle_commitment:
+  mode: batch_rebuild_from_data_at_rest
+  rebuild_every: 1m
+  retain_for: 10m
+
+backend_capabilities_and_evidence:
+  supported_modes: [batch_rebuild_from_data_at_rest]
+  supported_algorithms: [kll]
+  kll_200_state_bytes: 4096
+  five_minute_rebuild_cpu_ms: 35


I do not understand these. Is there documentation?

milindsrivastava1997 · 2026-09-18T13:33:33Z

+    - id: def-api-latency-kll
+      input: request_latency_seconds
+      group_by: [service]
+      range: 5m


Examples are helpful thank you. What does range mean?

range: 5m means the logical input window summarized by the KLL. For an evaluation at time T, it includes samples with timestamps in (T - 5m, T], grouped by service.

So basically it's equal to the size of the time window (tumbling or sliding) used to generate the KLL instances?

milindsrivastava1997 · 2026-09-18T13:36:09Z

-The baseline is merged code, not the completion of open PRs.
-
-| Area | Existing foundation | Consolidation needed |
+“Maintenance” is the execution phase that constructs or updates state, including


These concepts are also present in ASAPPlanner right? Wondering if these have been described there

milindsrivastava1997 · 2026-09-18T13:41:37Z

-A legal target alternative is:
+| Output | Responsibility |
+| --- | --- |
+| Catalog/SDS entries | Summary semantics, materialization identity, schema and state references |


How is this catalog actually used?

Catalog stores the semantics / description of SDS (changed less often), the summary store/sketch store stores the SDS instances payloads (changed per instance).

Catalog is metadata store

I understand what it is. I am curious, how it is used right now. Is it used right now?

catalog is being used rn.

How it's being used --

Compilation: The physical compiler constructs SummaryCatalog from the selected precompute configurations, binds PrecomputePlan to it, and validates QueryPlan against it. Compiler code

Runtime installation: When the startup physical plan contains a catalog, the data plane installs it into SketchStore. Startup code

Query execution: One concrete consumer is MetricsQL per-series readout: it looks up the bound definition in the catalog, resolves its data descriptor, and restores the metric’s name
label. If that metadata is unavailable, that path requests fallback. Readout code

The current catalog contains summary_descriptors, data_descriptors, and a materializations map. The simplified catalog described in this PR is a proposed change to that existing
representation—not the introduction of a previously unused catalog. Current type

milindsrivastava1997 · 2026-09-18T13:48:41Z

+| Object | Meaning | Changes when |
+| --- | --- | --- |
+| `SummaryDefinition` | Canonical input, operation, grouping, time semantics, algorithm and parameters | Summary semantics change |
+| `Materialization` | An installed decision to produce a definition with one state contract | Plan generation or physical contract changes |


Confused by this. I dont understand the meaning. Also, how is this "Materialization" related to the discussion at #736 (comment)

SummaryDefinition: summarize request_latency_seconds by service over five minutes using KLL with k=200.

SummaryStateInstance: one concrete stored result from that producer, such as the summary for service=api covering (12:00, 12:05].

Agreed—the standalone catalog Materialization object was an over-abstraction. I have removed it from the proposed design and examples rather than introducing another name for the same layer.

Its information now belongs to the objects that use it:

Information Owner

Summary meaning, state family, algorithm parameters SummaryDefinition in the catalog

Plan version The installed plan bundle; persisted instances also record it for recovery validation

Connection between writer and readers A compiler-assigned state_slot_id in their StateReference, scoped to the plan version

Schema/encoding and physical partition rules PrecomputePlan writer configuration and matching QueryPlan reader configuration

Authorized writer and maintenance policy The PrecomputePlan binding and selected producer lifecycle

Physical-to-semantic provenance The compiler's provenance mapping

Actual partition, coverage, readiness, location and payload format SummaryStateInstance runtime metadata; payload bytes remain in the summary store

The resulting flow is simply:

PrecomputePlan: Build KLL -> Write state slot S QueryPlan: Read state slot S -> Estimate p99 Catalog: SummaryDefinition referenced by both bindings

The state slot is only a join key within a plan version, not a new catalog object with an independent lifecycle. The compiler emits both bindings from one decision and validates their agreement before installation.

Regarding #736: BackendNodeBinding::Materialization remains the existing node-placement marker meaning “store this node's output.” It does not require a separate catalog Materialization object. I have made that distinction explicit.

The docs now remove the proposed materializations collection, update the diagrams and examples, and describe migration through versioned adapters: map existing stored-output identities to slots, retain payload locators, and preserve the format/partition constraints in reader/writer bindings. Existing persisted IDs and wire fields must not be silently renamed or reinterpreted. This PR remains a design-document change; runtime migration is follow-up implementation work.

Okay thanks

milindsrivastava1997 · 2026-09-18T13:49:52Z

+| `SummaryDefinition` | Canonical input, operation, grouping, time semantics, algorithm and parameters | Summary semantics change |
+| `Materialization` | An installed decision to produce a definition with one state contract | Plan generation or physical contract changes |
+| `SummaryStateInstance` | One stored partition, such as a series/pane or completed aggregate | Runtime creates or replaces payload state |
+| `StateReference` | A typed plan reference to permitted materialized state | A compiled reader/writer binding changes |


I did not understand the purpose of this.

statereference is how plan can reference the summarystateinstance.

milindsrivastava1997 · 2026-09-18T20:26:22Z

+| Selected post-ASAP DAG | Planner-selected computation graph, including summary producers, shared dependencies and query readouts. Called the “semantic DAG” in earlier discussion. |
+| Summary producer | An operation or subgraph that builds summary state. Multiple queries may share its stored output. |
+| `SummaryMaintenanceLifecyclePlan` | Planner result associating a post-ASAP root with deployment decisions for its unique reachable summary producers, plus workload and costing context. |
+| Lifecycle commitment | Selected maintenance promise for one producer, with its scheduling and retention binding. A deployment's `SummaryMaintenanceLifecycleGuarantee` carries the Planner-level commitment. |


Still confused on this. Why do we need this concept?

milindsrivastava1997 · 2026-09-18T20:29:19Z

+window framework. The plan also carries workload demand and costing context.
+Thus the lifecycle plan already refers to the computation DAG; it is not a
+separate query representation, nor is one whole lifecycle plan required per
+producer. A missing guarantee is not an executable maintenance commitment.


Did not understand last sentence on missing guarantee

milindsrivastava1997 · 2026-09-18T20:30:55Z

+physical plans plus catalog bindings. A shared producer is maintained once for
+all compatible consumers.
+
+### Lifecycle commitment


Very confused by this. Can you give an example situation where if I do not have this concept, there is some issue with correctness or performance or something else?

I saw the compiler input example, but still do not understand this concept.

milindsrivastava1997 · 2026-09-18T20:33:13Z

Workload example is helpful, thank you

milindsrivastava1997 · 2026-09-18T20:35:59Z

+There is no separate catalog `Materialization` object.

-The control plane reconciles two explicitly separate views:
+```mermaid


For the edges to SummaryDefinition catalog, which are reads and which are writes?
Related, are you saying that if I think of an SDS, the SummaryStore only stores SDS payload, while other metadata stays in the SummaryDefinition metadata catalog?

zzylol added 6 commits September 17, 2026 18:23

docs: clarify Planner physical plan and SDS architecture

ac30f32

docs: specify executable subplan materialization boundaries

af61296

docs: scope migration to backend precompute and query plans

e868424

docs: clarify window terminology migration

daa5281

Revert "docs: clarify window terminology migration"

e27bbb0

This reverts commit daa5281.

docs: focus physical plan and SDS designs

2230795

zzylol changed the title ~~docs: clarify Planner physical plan and SDS architecture~~ docs: define backend plan split and SDS state contract Sep 18, 2026

zzylol added 4 commits September 18, 2026 03:43

docs: add physical compiler input example

3f1830d

docs: add physical compiler output example

dd0a8b4

docs: include query expression in compiler example

c1fbfb6

docs: reorganize physical plan integration design

887c526

zzylol requested a review from milindsrivastava1997 September 18, 2026 04:03

docs: reorganize SDS and migration designs

b8193bb

milindsrivastava1997 reviewed Sep 18, 2026

View reviewed changes

zzylol added 4 commits September 18, 2026 16:37

docs: define maintenance inputs before plan split example

b24867b

docs: use plan version consistently in backend design

b358074

docs: remove standalone catalog materialization abstraction

7e4655a

docs: add concise planner backend glossary

07580f8

milindsrivastava1997 reviewed Sep 18, 2026

View reviewed changes

Boundary	Current implementation	Proposed integration
Computation supplied to compiler	Complete selected DAG root	Complete root retained in the lifecycle-aware result
Lifecycle selection	Called from inside physical compilation for extracted producers	Completed through Planner's helper before physical compilation
Connecting DAG and maintenance decisions	Backend combines the separate call results	Compiler consumes their association in the selected result
Compiler responsibility	Obtains lifecycle decisions and generates physical plans	Validates feasibility and generates physical plans from the selected decisions
Raw recomputation	Must be handled by the existing selection/compilation paths	Explicitly honor the helper's raw-versus-summary decision


		## Architecture at a glance

		The current `PrecomputePlan.executable_dags` can contain a complete semantic DAG,

Information	Owner
Summary meaning, state family, algorithm parameters	`SummaryDefinition` in the catalog
Plan version	The installed plan bundle; persisted instances also record it for recovery validation
Connection between writer and readers	A compiler-assigned `state_slot_id` in their `StateReference`, scoped to the plan version
Schema/encoding and physical partition rules	PrecomputePlan writer configuration and matching QueryPlan reader configuration
Authorized writer and maintenance policy	The PrecomputePlan binding and selected producer lifecycle
Physical-to-semantic provenance	The compiler's provenance mapping
Actual partition, coverage, readiness, location and payload format	`SummaryStateInstance` runtime metadata; payload bytes remain in the summary store

Conversation

zzylol commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zzylol Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Current implementation

Proposed integration

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zzylol commented Sep 17, 2026 •

edited

Loading

zzylol Sep 18, 2026 •

edited

Loading