From e71a6c43b083e7786a3cfe51e7a8780669bd41b5 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 18:38:40 +0000 Subject: [PATCH 01/22] docs: define planner input output contract --- docs/design_docs/README.md | 3 + docs/design_docs/architecture/README.md | 4 + .../architecture/planner-contract.md | 310 ++++++++++++++++++ 3 files changed, 317 insertions(+) create mode 100644 docs/design_docs/architecture/planner-contract.md diff --git a/docs/design_docs/README.md b/docs/design_docs/README.md index 1b9f8ff4..6a99b200 100644 --- a/docs/design_docs/README.md +++ b/docs/design_docs/README.md @@ -7,6 +7,9 @@ decisions, and evolving proposals. and Pre-ASAP/Post-ASAP representations. - [Architecture](architecture/README.md) describes stable component boundaries and system flows. +- [Planner input, output, and workflows](architecture/planner-contract.md) + defines the user-facing contract, required and optional inputs, output data + structures, and initial-planning and replanning workflows. - [Decisions](decisions/README.md) records accepted local design choices. - [Proposals](proposals/README.md) contains evolving designs and research directions; check each document’s status before treating it as current. diff --git a/docs/design_docs/architecture/README.md b/docs/design_docs/architecture/README.md index a43215b7..5e76bdc6 100644 --- a/docs/design_docs/architecture/README.md +++ b/docs/design_docs/architecture/README.md @@ -6,6 +6,10 @@ It does not commit, deploy, or execute a physical plan; downstream systems such as ASAPQuery-backend bind the candidates to physical alternatives, make the deployment-level decision, and run the selected contract. +For the integration contract, start with [ASAPPlanner input, output, and +workflows](planner-contract.md). It defines required and optional inputs, +`PlanSpace` and selected-plan outputs, lifecycle requirements, and replanning. + ## Planner component flow ```mermaid diff --git a/docs/design_docs/architecture/planner-contract.md b/docs/design_docs/architecture/planner-contract.md new file mode 100644 index 00000000..d7b298d2 --- /dev/null +++ b/docs/design_docs/architecture/planner-contract.md @@ -0,0 +1,310 @@ +# ASAPPlanner input, output, and workflows + +## Purpose + +This document defines the public mental model for embedding ASAPPlanner. It +answers three questions: + +1. What must a caller provide? +2. What does Planner return? +3. Which workflow should a caller use? + +ASAPPlanner is a **logical planning library**. It accepts canonical query roots +and their requirements, explores legal ways to answer them with exact or +approximate summaries, and returns the resulting candidate space. It does not +deploy or execute a plan. + +The central contract is: + +```text +canonical query roots + requirements + optional planning context + | + v + ASAPPlanner + | + v + PlanSpace: legal Post-ASAP alternatives +``` + +Candidate discovery, local replacement strategies, memo groups, accuracy +allocation, and ranking are internal stages of this contract. They are public +Rust extension points, but an application does not need to call them one by one. + +## The boundary in one example + +For a recurring PromQL quantile query, the caller: + +1. lowers the source query to a canonical Pre-ASAP `QueryExpr`; +2. associates that root with an application query ID and accuracy requirement; +3. asks Planner to search the workload; and +4. receives a `PlanSpace` containing the exact alternative and every proven + legal summary alternative. + +A downstream system may inspect all alternatives or ask Planner's selection +helpers to choose and materialize a Post-ASAP DAG. The downstream system still +binds that logical DAG to executable operators, storage, and placement. + +If a proof or cost is missing, Planner does not invent it. The affected +optimization is absent, rejected, or ranked as unavailable; an exact +`KeepPreAsap` alternative preserves the original computation where supported. + +## Input contract + +There are two input layers. A frontend translates source-language input into +the canonical input accepted by the planning core. + +### Source-language input + +| Input | Required | Meaning if omitted or unknown | +|---|---:|---| +| Query text | Yes | There is no query to plan. | +| Query language | Yes | The caller must select the matching frontend. | +| Schema/function catalog | Frontend-dependent | SQL name and type resolution fails without the required catalog. | +| Per-query accuracy requirement | Yes | Use `Exact` explicitly when approximation is not allowed. | +| Query recurrence and time selection | Required for lifecycle-aware comparison | Candidate discovery can proceed, but Planner cannot compare repeated maintenance with raw recomputation over time. | +| Data ingestion interval | Required by PromQL workload lowering | Bare selectors have no defensible selection horizon without it. | +| Other data-workload facts | Optional | Optimizations requiring an unknown fact remain unavailable; unknown never means zero. | + +The normalized workload types are `PlanningWorkload`, `QueryWorkload`, and +optional `DataWorkload`. Frontend output is one Pre-ASAP `QueryExpr` root per +workload entry. The caller must retain the association between each root and +its workload entry. + +### Canonical planning input + +The planning core consumes: + +```text +Vec<(query_id, Rc, Option)> +``` + +The fields have these roles: + +| Field | Required | Role | +|---|---:|---| +| `query_id` | Yes | Caller-owned identity used to associate results with queries. | +| `QueryExpr` | Yes | Canonical exact query semantics; this is the Pre-ASAP root. | +| Root `AccuracyTarget` | Recommended; required for an enforced end-to-end target | Filters out summary candidates whose final guarantee is missing or insufficient. `None` means that this call adds no root-level requirement. | +| Strategy set | Yes in the configurable API | Use `default_strategies_with_evidence` for the standard set with deployment models. Custom sets are an extension mechanism, not separate workflow stages. | +| `AccuracyModel` | Yes in the configurable API | Defines how guarantees compose and whether they satisfy a target. Most callers use `DefaultAccuracyModel`. | +| `CostModel` | Required for ranking or selection | Reports candidate availability and preference. The built-in default is useful for inspection, not a claim about deployment cost. | +| `AccuracyEvidenceProvider` | Optional in general; required by candidates whose proof needs it | Supplies typed facts such as certified quantile input domains or Top-K separation. Without a required proof, that candidate fails closed. | + +### Inputs used for lifecycle-aware selection + +Lifecycle-aware selection answers a narrower question: is it cheaper to build, +maintain, and read a summary, or to execute the raw query for the declared +workload? + +| Input | Required for lifecycle-aware selection | Role | +|---|---:|---| +| Query workload binding | Yes | Identifies which workload entries consume each root. | +| Planning time (`now_ms`) | Yes | Evaluates schedules and evidence freshness. | +| Planning horizon | Required for finite recurring totals | Bounds the number of reads and updates. Missing horizon leaves some totals unknown. | +| Data arrival and update rate | Required for continuously maintained cost | Prices maintenance work. Unknown values cannot be treated as no updates. | +| Lifecycle capabilities | Yes | Declares which lifecycle alternatives the deployment can implement. | +| Complete comparable cost inputs | Required to choose on cost | Both the summary path and raw baseline must be costed in the same scope. | + +Lifecycle information is therefore not an optional accuracy check. It is +optional only when the caller wants candidate discovery or a structural +selection rather than a workload-costed deployment decision. + +## What “evidence” means + +**Evidence is a scoped fact used to justify candidate legality, accuracy, or +cost.** It may be declared by a contract, derived analytically, measured in an +offline benchmark, or observed online. Evidence must identify the workload and +implementation to which it applies and, when time-sensitive, carry freshness +information. + +| Kind | Examples | Consequence when required but missing | +|---|---|---| +| Semantic or domain evidence | Finite input range, nonempty population, denominator excludes zero | The transformation cannot prove its preconditions. | +| Accuracy evidence | Quantile domain, Top-K confidence margins, composition certificate | The approximate candidate has no valid end-to-end guarantee. | +| Cost evidence | CPU time, operation count, retained/peak bytes, scan or storage I/O | Planner cannot make the corresponding cost comparison. | +| Workload evidence | Cardinality, distribution, ingestion rate, recurrence | Dependent sizing, propagation, or lifecycle costs remain unknown. | +| Capability evidence | Supported summary, deletion, merge, or window framework | A physically unsupported alternative is unavailable. | + +One fact may support more than one calculation. Cardinality, for example, may +affect both an accuracy bound and a resource estimate. The consumer determines +its role; the word “evidence” does not mean “cost measurement.” Historical +observations also do not prove a permanent input-domain invariant unless the +named contract enforces that invariant for the plan's lifetime. + +## Output contract + +### Canonical output: `PlanSpace` + +`PlanSpace` is the complete logical output of search. It contains: + +- the workload roots after canonical common-subexpression sharing; +- one `MemoGroup` for every discovered target sub-DAG; +- every legal replacement candidate retained for that target; +- rejected candidates and their reasons; and +- prepared cross-group composition information used by selection. + +The output is a **space of DAG choices**, not one executable DAG. A memo group +is a decision point for one canonical subexpression, not a standalone workload +plan. Choosing the first candidate independently in every group is not a valid +substitute for whole-plan selection because sharing and composition decisions +can change downstream uses. + +`PlanSpace::cost_sorted` is a read-only ranked view: + +```text +Vec +``` + +It is intended for inspection, explanation, or a downstream physical planner +that must retain alternatives. A listed candidate is logically available; its +presence alone does not prove that the deployment can execute it. + +### Selected output: `GlobalSelection` and materialized DAGs + +`PlanSpace::global_selection*` coordinates choices across memo groups. +`GlobalSelection::materialize(root)` returns the chosen `Rc` +Post-ASAP DAG for that root. A workload therefore produces one materialized +root per input query ID, with shared `Rc` nodes where the selection shares +state. + +The materialized DAG records logical semantics, including summary family and +parameters, operations, schemas, windows, and result guarantees. It is not yet +an executable deployment plan. A downstream compiler must bind it to supported +physical operators and may preserve `KeepPreAsap` regions for exact execution. + +### Lifecycle-aware selected output + +`SummaryMaintenanceLifecyclePlan` adds the information needed to explain a +workload-aware maintenance decision: + +- the materialized root; +- selected or rejected lifecycle alternatives for each summary state; +- horizon, evaluation rate, update rate, and expected reads; +- selected window implementation identity and accuracy guarantee, when known; +- summary and raw-recompute costs, when comparable; and +- whether raw recomputation was selected. + +This remains a planner contract, not runtime configuration. The downstream +system compiles, deploys, and executes it. + +## Supported workflows + +### Workflow 1: inspect all logical alternatives + +Use this workflow for tooling, explanations, or a downstream optimizer that +performs its own physical comparison. + +```text +PlanningWorkload + -> frontend lowering + -> search_workload_with_targets + -> PlanSpace + -> cost_sorted (optional view) +``` + +Promise: all alternatives retained by the configured semantic, accuracy, and +evidence checks are visible. + +Does not promise: one committed plan, complete physical feasibility, calibrated +deployment cost, or a lifecycle decision. + +### Workflow 2: select a logical DAG + +Use this workflow when the caller wants Planner to coordinate sharing and +composition choices but does not need a maintenance-versus-recompute decision. + +```text +PlanSpace + -> global_selection + -> materialize each workload root + -> selected Post-ASAP DAGs +``` + +Promise: choices are structurally compatible across the logical workload. + +Does not promise: that the selected summary has a deployable physical +implementation or that maintaining it is cheaper than raw execution. + +### Workflow 3: make a lifecycle-aware planning decision + +This is the recommended workflow for a downstream system deciding whether to +deploy maintained summary state. + +```text +PlanningWorkload + Pre-ASAP roots + -> search_workload_with_targets + -> global_selection_with_summary_maintenance_lifecycles + -> materialize_with_summary_maintenance_lifecycles + -> lifecycle-aware Post-ASAP plans + -> downstream physical binding and deployment +``` + +Promise: the selection uses declared recurrence, data arrival, horizon, +capabilities, and comparable raw/summary costs. Missing required facts fail +closed instead of becoming optimistic zeroes. + +Does not promise: placement, cluster-capacity feasibility, runtime readiness, +or execution. Those remain downstream responsibilities. + +## Replanning + +Replanning uses the same contract as initial planning. There is no separate +mutable Planner session whose hidden state changes the answer. + +The caller should invoke the workflow again when any selection-relevant input +changes, including: + +- query text, schema, or accuracy requirement; +- recurrence, horizon, data arrival, or distribution; +- evidence expiration or replacement; +- supported physical capabilities; +- cost calibration; or +- available materialized state. + +```text +new workload snapshot + new evidence/capabilities + | + v + run planning again + | + v + new PlanSpace / selected contracts + | + v + downstream compares, transitions, and activates +``` + +Planner produces the new logical decision. The downstream system owns diffing +old and new deployments, migration, activation ordering, and rollback. Evidence +from one planning snapshot must not be silently reused after its validity or +comparison scope changes. + +## API guidance + +For normal integrations: + +- lower the complete workload through one frontend; +- call a whole-workload `search_workload*` function once; +- use the standard strategy factory rather than invoking individual strategies; +- preserve explicit accuracy targets and required evidence; +- use lifecycle-aware selection before claiming that a maintained summary is + preferable to exact recomputation; and +- treat physical compilation and deployment as a downstream step. + +The lower-level public traits and functions support research and deployment +extensions. They are not additional mandatory stages and should not be +presented as independent end-user workflows. + +## Related documents + +- [Planner pipeline](../concepts/planner-pipeline.md) +- [Pre-ASAP IR](../concepts/pre-asap-ir.md) +- [Post-ASAP IR](../concepts/post-asap-ir.md) +- [Planner/downstream boundary](planner-downstream-boundary.md) +- [Plan search internals](asap-aware-plan-search.md) +- [Public library reference](../../develop_docs/library-api.md) From b4841b364ca011745ec6497a63d71eaf49ba9121 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 18:41:29 +0000 Subject: [PATCH 02/22] docs: clarify planner and runtime vocabulary --- docs/design_docs/README.md | 2 +- docs/design_docs/architecture/README.md | 2 +- ...r-contract.md => input-output-workflow.md} | 41 ++++++++++++++++++- 3 files changed, 42 insertions(+), 3 deletions(-) rename docs/design_docs/architecture/{planner-contract.md => input-output-workflow.md} (82%) diff --git a/docs/design_docs/README.md b/docs/design_docs/README.md index 6a99b200..23a789d2 100644 --- a/docs/design_docs/README.md +++ b/docs/design_docs/README.md @@ -7,7 +7,7 @@ decisions, and evolving proposals. and Pre-ASAP/Post-ASAP representations. - [Architecture](architecture/README.md) describes stable component boundaries and system flows. -- [Planner input, output, and workflows](architecture/planner-contract.md) +- [Planner input, output, and workflow](architecture/input-output-workflow.md) defines the user-facing contract, required and optional inputs, output data structures, and initial-planning and replanning workflows. - [Decisions](decisions/README.md) records accepted local design choices. diff --git a/docs/design_docs/architecture/README.md b/docs/design_docs/architecture/README.md index 5e76bdc6..0f7916d4 100644 --- a/docs/design_docs/architecture/README.md +++ b/docs/design_docs/architecture/README.md @@ -7,7 +7,7 @@ as ASAPQuery-backend bind the candidates to physical alternatives, make the deployment-level decision, and run the selected contract. For the integration contract, start with [ASAPPlanner input, output, and -workflows](planner-contract.md). It defines required and optional inputs, +workflow](input-output-workflow.md). It defines required and optional inputs, `PlanSpace` and selected-plan outputs, lifecycle requirements, and replanning. ## Planner component flow diff --git a/docs/design_docs/architecture/planner-contract.md b/docs/design_docs/architecture/input-output-workflow.md similarity index 82% rename from docs/design_docs/architecture/planner-contract.md rename to docs/design_docs/architecture/input-output-workflow.md index d7b298d2..f148b678 100644 --- a/docs/design_docs/architecture/planner-contract.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -1,4 +1,4 @@ -# ASAPPlanner input, output, and workflows +# ASAPPlanner input, output, and workflow ## Purpose @@ -131,6 +131,38 @@ its role; the word “evidence” does not mean “cost measurement.” Historic observations also do not prove a permanent input-domain invariant unless the named contract enforces that invariant for the plan's lifetime. +## Vocabulary across Planner and runtime + +The input and output workflow uses different terms for different decisions. +They must not be collapsed into one generic “window,” “implementation,” or +“boundary” concept. + +| Term | Owner | Meaning | +|---|---|---| +| Query window | Query semantics | The interval requested by the query, such as the five minutes in `data[5m]`. | +| Evaluation cadence or slide | Workload semantics | When the query is evaluated; it does not say how state is stored. | +| Summary window framework | ASAPPlanner | The abstract algorithm for organizing maintained summary state: tumbling, sliding, exponential histogram, or a registered extension. | +| Physical window layout | Downstream runtime | The concrete storage organization, such as full-window states, fixed panes, or hierarchical pane rollups. | +| Pane layout | Planner-runtime interface | Pane width and phase/origin needed to prove exact temporal coverage. A pane is a disjoint stored interval, not the query window itself. | +| Window-edge coverage | Planner-runtime interface | How partial intervals at a query window's edges are answered, for example by alignment or an exact residual. | +| Physical handoff | Physical costing/runtime | A network transfer or intermediate materialization. It is unrelated to a query-window edge. | +| Comparison scope | Costing | The common source, predicates, time selection, recurrence, and horizon over which raw and summary costs are comparable. | + +These concepts may map to enums in Planner or a downstream repository, but an +enum is justified only when its variants express a live decision at that +owner's layer. For example, Planner's `SummaryWindowFramework` describes an +abstract choice it can compare; a backend physical-layout enum describes how +that selected choice is stored. A second enum that repeats the same decision +without adding an ownership or translation boundary should be removed or kept +internal. + +Use **realization** for a candidate physical form and reserve +**implementation** for executable code. Use **schema resolution** for resolving +names and types, **window edge** for temporal coverage, **physical handoff** for +transfer or materialization, and **comparison scope** for cost comparability. +These names keep the workflow understandable while compatibility aliases remain +in code. + ## Output contract ### Canonical output: `PlanSpace` @@ -300,6 +332,13 @@ The lower-level public traits and functions support research and deployment extensions. They are not additional mandatory stages and should not be presented as independent end-user workflows. +Public Rust visibility does not by itself make a type part of the recommended +integration surface. New public enums, variants, and extension points require +a concrete workflow that consumes them. API review should remove or internalize +duplicate concepts, unused variants, and compatibility types after their +consumers have migrated. This document names the intended external concepts; +the library reference records the current Rust entry points. + ## Related documents - [Planner pipeline](../concepts/planner-pipeline.md) From c4cae3d2bbe21b4a8e2069e5185b0be932020595 Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Fri, 18 Sep 2026 14:44:40 -0400 Subject: [PATCH 03/22] Update input-output-workflow.md --- .../architecture/input-output-workflow.md | 447 +++++++----------- 1 file changed, 182 insertions(+), 265 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index f148b678..209fc73e 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -1,349 +1,266 @@ -# ASAPPlanner input, output, and workflow +# ASAPPlanner input, output, and workflows -## Purpose +## Overview -This document defines the public mental model for embedding ASAPPlanner. It -answers three questions: +ASAPPlanner is a **logical planning library**. It takes canonical queries and their requirements, explores legal exact and approximate implementations, and returns a logical plan space. -1. What must a caller provide? -2. What does Planner return? -3. Which workflow should a caller use? +It does **not** deploy or execute plans. -ASAPPlanner is a **logical planning library**. It accepts canonical query roots -and their requirements, explores legal ways to answer them with exact or -approximate summaries, and returns the resulting candidate space. It does not -deploy or execute a plan. +```text +canonical queries + requirements + planning context + | + v + ASAPPlanner + | + v + PlanSpace + (legal Post-ASAP alternatives) +``` + +A typical integration: + +1. Lower source queries into canonical Pre-ASAP `QueryExpr` roots. +2. Attach query IDs and accuracy requirements. +3. Run Planner over the workload. +4. Inspect the resulting alternatives or select a Post-ASAP DAG. +5. Bind the selected logical DAG to physical operators downstream. + +If required accuracy, semantic, capability, or cost evidence is unavailable, Planner does not assume it. Unsupported optimizations fail closed, while `KeepPreAsap` preserves exact computation where supported. + +--- + +## Inputs + +### Source-language inputs + +A frontend converts source-language queries into Planner's canonical representation. -The central contract is: +| Input | Required | Purpose | +| ------------------------------- | ---------------------: | -------------------------------------------------------- | +| Query text | Yes | Defines the query to plan. | +| Query language | Yes | Selects the appropriate frontend. | +| Schema/function catalog | Frontend-dependent | Resolves names and types. | +| Accuracy requirement | Yes | Use `Exact` when approximation is not allowed. | +| Query recurrence/time selection | For lifecycle planning | Describes how often and over what period the query runs. | +| Data ingestion interval | For PromQL lowering | Defines the selection horizon for bare selectors. | +| Other workload facts | Optional | Enable optimizations that depend on them. | + +Frontend lowering produces one Pre-ASAP `QueryExpr` root for each workload query. + +### Canonical Planner inputs + +The planning core operates on: ```text -canonical query roots + requirements + optional planning context - | - v - ASAPPlanner - | - v - PlanSpace: legal Post-ASAP alternatives +Vec<(query_id, Rc, Option)> ``` -Candidate discovery, local replacement strategies, memo groups, accuracy -allocation, and ranking are internal stages of this contract. They are public -Rust extension points, but an application does not need to call them one by one. +The main inputs are: -## The boundary in one example +* **`query_id`** — caller-owned query identity. +* **`QueryExpr`** — canonical exact query semantics. +* **`AccuracyTarget`** — required end-to-end accuracy, if enforced. +* **Strategies** — candidate transformations Planner may explore. +* **`AccuracyModel`** — determines how guarantees compose and satisfy targets. +* **`CostModel`** — evaluates candidate cost and availability. +* **`AccuracyEvidenceProvider`** — provides facts required to prove candidate guarantees. -For a recurring PromQL quantile query, the caller: +Most integrations should use the standard strategy set and `DefaultAccuracyModel`. -1. lowers the source query to a canonical Pre-ASAP `QueryExpr`; -2. associates that root with an application query ID and accuracy requirement; -3. asks Planner to search the workload; and -4. receives a `PlanSpace` containing the exact alternative and every proven - legal summary alternative. +### Additional inputs for lifecycle planning -A downstream system may inspect all alternatives or ask Planner's selection -helpers to choose and materialize a Post-ASAP DAG. The downstream system still -binds that logical DAG to executable operators, storage, and placement. +Lifecycle-aware planning compares maintaining a summary against recomputing the raw query. -If a proof or cost is missing, Planner does not invent it. The affected -optimization is absent, rejected, or ranked as unavailable; an exact -`KeepPreAsap` alternative preserves the original computation where supported. +It additionally requires: -## Input contract +* query workload bindings; +* planning time and horizon; +* query recurrence; +* data arrival/update rate; +* deployment lifecycle capabilities; and +* comparable cost information for summary and raw execution. -There are two input layers. A frontend translates source-language input into -the canonical input accepted by the planning core. +Missing required information remains **unknown** rather than being treated as zero. -### Source-language input +--- -| Input | Required | Meaning if omitted or unknown | -|---|---:|---| -| Query text | Yes | There is no query to plan. | -| Query language | Yes | The caller must select the matching frontend. | -| Schema/function catalog | Frontend-dependent | SQL name and type resolution fails without the required catalog. | -| Per-query accuracy requirement | Yes | Use `Exact` explicitly when approximation is not allowed. | -| Query recurrence and time selection | Required for lifecycle-aware comparison | Candidate discovery can proceed, but Planner cannot compare repeated maintenance with raw recomputation over time. | -| Data ingestion interval | Required by PromQL workload lowering | Bare selectors have no defensible selection horizon without it. | -| Other data-workload facts | Optional | Optimizations requiring an unknown fact remain unavailable; unknown never means zero. | +## Evidence -The normalized workload types are `PlanningWorkload`, `QueryWorkload`, and -optional `DataWorkload`. Frontend output is one Pre-ASAP `QueryExpr` root per -workload entry. The caller must retain the association between each root and -its workload entry. +**Evidence is a scoped fact used to establish legality, accuracy, cost, or feasibility.** -### Canonical planning input +Examples include: -The planning core consumes: +| Evidence | Examples | +| --------------- | ---------------------------------------------------------------- | +| Semantic/domain | Input range, nonempty population, nonzero denominator | +| Accuracy | Quantile domain, Top-K confidence, composition certificate | +| Cost | CPU time, operation count, memory, scan/storage I/O | +| Workload | Cardinality, distribution, ingestion rate, recurrence | +| Capability | Supported summaries, merge/delete support, window implementation | -```text -Vec<(query_id, Rc, Option)> -``` +Evidence must apply to the relevant workload and implementation. Time-sensitive evidence should also carry freshness information. + +When required evidence is missing, the dependent optimization is unavailable. -The fields have these roles: - -| Field | Required | Role | -|---|---:|---| -| `query_id` | Yes | Caller-owned identity used to associate results with queries. | -| `QueryExpr` | Yes | Canonical exact query semantics; this is the Pre-ASAP root. | -| Root `AccuracyTarget` | Recommended; required for an enforced end-to-end target | Filters out summary candidates whose final guarantee is missing or insufficient. `None` means that this call adds no root-level requirement. | -| Strategy set | Yes in the configurable API | Use `default_strategies_with_evidence` for the standard set with deployment models. Custom sets are an extension mechanism, not separate workflow stages. | -| `AccuracyModel` | Yes in the configurable API | Defines how guarantees compose and whether they satisfy a target. Most callers use `DefaultAccuracyModel`. | -| `CostModel` | Required for ranking or selection | Reports candidate availability and preference. The built-in default is useful for inspection, not a claim about deployment cost. | -| `AccuracyEvidenceProvider` | Optional in general; required by candidates whose proof needs it | Supplies typed facts such as certified quantile input domains or Top-K separation. Without a required proof, that candidate fails closed. | - -### Inputs used for lifecycle-aware selection - -Lifecycle-aware selection answers a narrower question: is it cheaper to build, -maintain, and read a summary, or to execute the raw query for the declared -workload? - -| Input | Required for lifecycle-aware selection | Role | -|---|---:|---| -| Query workload binding | Yes | Identifies which workload entries consume each root. | -| Planning time (`now_ms`) | Yes | Evaluates schedules and evidence freshness. | -| Planning horizon | Required for finite recurring totals | Bounds the number of reads and updates. Missing horizon leaves some totals unknown. | -| Data arrival and update rate | Required for continuously maintained cost | Prices maintenance work. Unknown values cannot be treated as no updates. | -| Lifecycle capabilities | Yes | Declares which lifecycle alternatives the deployment can implement. | -| Complete comparable cost inputs | Required to choose on cost | Both the summary path and raw baseline must be costed in the same scope. | - -Lifecycle information is therefore not an optional accuracy check. It is -optional only when the caller wants candidate discovery or a structural -selection rather than a workload-costed deployment decision. - -## What “evidence” means - -**Evidence is a scoped fact used to justify candidate legality, accuracy, or -cost.** It may be declared by a contract, derived analytically, measured in an -offline benchmark, or observed online. Evidence must identify the workload and -implementation to which it applies and, when time-sensitive, carry freshness -information. - -| Kind | Examples | Consequence when required but missing | -|---|---|---| -| Semantic or domain evidence | Finite input range, nonempty population, denominator excludes zero | The transformation cannot prove its preconditions. | -| Accuracy evidence | Quantile domain, Top-K confidence margins, composition certificate | The approximate candidate has no valid end-to-end guarantee. | -| Cost evidence | CPU time, operation count, retained/peak bytes, scan or storage I/O | Planner cannot make the corresponding cost comparison. | -| Workload evidence | Cardinality, distribution, ingestion rate, recurrence | Dependent sizing, propagation, or lifecycle costs remain unknown. | -| Capability evidence | Supported summary, deletion, merge, or window framework | A physically unsupported alternative is unavailable. | - -One fact may support more than one calculation. Cardinality, for example, may -affect both an accuracy bound and a resource estimate. The consumer determines -its role; the word “evidence” does not mean “cost measurement.” Historical -observations also do not prove a permanent input-domain invariant unless the -named contract enforces that invariant for the plan's lifetime. - -## Vocabulary across Planner and runtime - -The input and output workflow uses different terms for different decisions. -They must not be collapsed into one generic “window,” “implementation,” or -“boundary” concept. - -| Term | Owner | Meaning | -|---|---|---| -| Query window | Query semantics | The interval requested by the query, such as the five minutes in `data[5m]`. | -| Evaluation cadence or slide | Workload semantics | When the query is evaluated; it does not say how state is stored. | -| Summary window framework | ASAPPlanner | The abstract algorithm for organizing maintained summary state: tumbling, sliding, exponential histogram, or a registered extension. | -| Physical window layout | Downstream runtime | The concrete storage organization, such as full-window states, fixed panes, or hierarchical pane rollups. | -| Pane layout | Planner-runtime interface | Pane width and phase/origin needed to prove exact temporal coverage. A pane is a disjoint stored interval, not the query window itself. | -| Window-edge coverage | Planner-runtime interface | How partial intervals at a query window's edges are answered, for example by alignment or an exact residual. | -| Physical handoff | Physical costing/runtime | A network transfer or intermediate materialization. It is unrelated to a query-window edge. | -| Comparison scope | Costing | The common source, predicates, time selection, recurrence, and horizon over which raw and summary costs are comparable. | - -These concepts may map to enums in Planner or a downstream repository, but an -enum is justified only when its variants express a live decision at that -owner's layer. For example, Planner's `SummaryWindowFramework` describes an -abstract choice it can compare; a backend physical-layout enum describes how -that selected choice is stored. A second enum that repeats the same decision -without adding an ownership or translation boundary should be removed or kept -internal. - -Use **realization** for a candidate physical form and reserve -**implementation** for executable code. Use **schema resolution** for resolving -names and types, **window edge** for temporal coverage, **physical handoff** for -transfer or materialization, and **comparison scope** for cost comparability. -These names keep the workflow understandable while compatibility aliases remain -in code. - -## Output contract - -### Canonical output: `PlanSpace` - -`PlanSpace` is the complete logical output of search. It contains: - -- the workload roots after canonical common-subexpression sharing; -- one `MemoGroup` for every discovered target sub-DAG; -- every legal replacement candidate retained for that target; -- rejected candidates and their reasons; and -- prepared cross-group composition information used by selection. - -The output is a **space of DAG choices**, not one executable DAG. A memo group -is a decision point for one canonical subexpression, not a standalone workload -plan. Choosing the first candidate independently in every group is not a valid -substitute for whole-plan selection because sharing and composition decisions -can change downstream uses. - -`PlanSpace::cost_sorted` is a read-only ranked view: +--- + +## Outputs + +### `PlanSpace` + +`PlanSpace` is Planner's canonical output. It contains: + +* canonical workload roots; +* memo groups for discovered target sub-DAGs; +* legal replacement candidates; +* rejected candidates and reasons; and +* information needed for cross-group selection. + +A `PlanSpace` represents a **space of logical DAG choices**, not a single executable plan. + +`PlanSpace::cost_sorted` provides a ranked view for inspection: ```text Vec ``` -It is intended for inspection, explanation, or a downstream physical planner -that must retain alternatives. A listed candidate is logically available; its -presence alone does not prove that the deployment can execute it. +This view is useful for debugging, explanation, or downstream optimization. Candidate presence does not imply physical deployability. + +### Selected Post-ASAP DAG + +`PlanSpace::global_selection*` coordinates decisions across memo groups. -### Selected output: `GlobalSelection` and materialized DAGs +```text +PlanSpace + | + v +global_selection + | + v +materialize(root) + | + v +Post-ASAP DAG +``` -`PlanSpace::global_selection*` coordinates choices across memo groups. -`GlobalSelection::materialize(root)` returns the chosen `Rc` -Post-ASAP DAG for that root. A workload therefore produces one materialized -root per input query ID, with shared `Rc` nodes where the selection shares -state. +The resulting DAG records logical information such as summary operators, parameters, schemas, windows, and accuracy guarantees. -The materialized DAG records logical semantics, including summary family and -parameters, operations, schemas, windows, and result guarantees. It is not yet -an executable deployment plan. A downstream compiler must bind it to supported -physical operators and may preserve `KeepPreAsap` regions for exact execution. +It is still **not an executable deployment plan**. Physical operator binding, placement, storage, and execution remain downstream responsibilities. -### Lifecycle-aware selected output +### Lifecycle-aware output -`SummaryMaintenanceLifecyclePlan` adds the information needed to explain a -workload-aware maintenance decision: +`SummaryMaintenanceLifecyclePlan` additionally records: -- the materialized root; -- selected or rejected lifecycle alternatives for each summary state; -- horizon, evaluation rate, update rate, and expected reads; -- selected window implementation identity and accuracy guarantee, when known; -- summary and raw-recompute costs, when comparable; and -- whether raw recomputation was selected. +* the materialized root; +* lifecycle choices for summary state; +* planning horizon and expected reads/updates; +* selected window implementation and guarantees; +* comparable summary and raw-recomputation costs; and +* whether raw recomputation was selected. -This remains a planner contract, not runtime configuration. The downstream -system compiles, deploys, and executes it. +--- -## Supported workflows +## Workflows -### Workflow 1: inspect all logical alternatives +### 1. Inspect logical alternatives -Use this workflow for tooling, explanations, or a downstream optimizer that -performs its own physical comparison. +Use when the caller wants to inspect Planner's candidate space or perform physical optimization downstream. ```text PlanningWorkload - -> frontend lowering - -> search_workload_with_targets - -> PlanSpace - -> cost_sorted (optional view) + -> frontend lowering + -> search_workload_with_targets + -> PlanSpace + -> cost_sorted (optional) ``` -Promise: all alternatives retained by the configured semantic, accuracy, and -evidence checks are visible. - -Does not promise: one committed plan, complete physical feasibility, calibrated -deployment cost, or a lifecycle decision. +This exposes legal logical alternatives but does not choose a deployment. -### Workflow 2: select a logical DAG +### 2. Select a logical DAG -Use this workflow when the caller wants Planner to coordinate sharing and -composition choices but does not need a maintenance-versus-recompute decision. +Use when Planner should coordinate sharing and composition across the workload. ```text PlanSpace - -> global_selection - -> materialize each workload root - -> selected Post-ASAP DAGs + -> global_selection + -> materialize + -> selected Post-ASAP DAGs ``` -Promise: choices are structurally compatible across the logical workload. +This produces structurally compatible logical plans. It does not determine whether maintaining summaries is cheaper than raw execution. -Does not promise: that the selected summary has a deployable physical -implementation or that maintaining it is cheaper than raw execution. +### 3. Make a lifecycle-aware decision -### Workflow 3: make a lifecycle-aware planning decision - -This is the recommended workflow for a downstream system deciding whether to -deploy maintained summary state. +Use when deciding whether maintained summary state should actually be deployed. ```text PlanningWorkload + Pre-ASAP roots - -> search_workload_with_targets - -> global_selection_with_summary_maintenance_lifecycles - -> materialize_with_summary_maintenance_lifecycles - -> lifecycle-aware Post-ASAP plans - -> downstream physical binding and deployment + -> search_workload_with_targets + -> global_selection_with_summary_maintenance_lifecycles + -> materialize_with_summary_maintenance_lifecycles + -> lifecycle-aware Post-ASAP plans + -> downstream physical deployment ``` -Promise: the selection uses declared recurrence, data arrival, horizon, -capabilities, and comparable raw/summary costs. Missing required facts fail -closed instead of becoming optimistic zeroes. +This workflow considers recurrence, data arrival, planning horizon, capabilities, and comparable summary/raw costs. + +It is the recommended workflow for deployment decisions. -Does not promise: placement, cluster-capacity feasibility, runtime readiness, -or execution. Those remain downstream responsibilities. +--- ## Replanning -Replanning uses the same contract as initial planning. There is no separate -mutable Planner session whose hidden state changes the answer. +Replanning uses the same interface as initial planning. -The caller should invoke the workflow again when any selection-relevant input -changes, including: +Run Planner again whenever a selection-relevant input changes, such as: -- query text, schema, or accuracy requirement; -- recurrence, horizon, data arrival, or distribution; -- evidence expiration or replacement; -- supported physical capabilities; -- cost calibration; or -- available materialized state. +* query semantics or accuracy requirements; +* recurrence, horizon, data arrival, or distribution; +* evidence; +* supported capabilities; +* cost calibration; or +* available materialized state. ```text -new workload snapshot + new evidence/capabilities +updated workload + evidence + capabilities | v - run planning again + run Planner | v - new PlanSpace / selected contracts + new PlanSpace / selection | v - downstream compares, transitions, and activates + downstream deployment transition ``` -Planner produces the new logical decision. The downstream system owns diffing -old and new deployments, migration, activation ordering, and rollback. Evidence -from one planning snapshot must not be silently reused after its validity or -comparison scope changes. +Planner produces a new logical decision. The downstream system owns deployment diffing, migration, activation, and rollback. -## API guidance +--- + +## Integration guidance For normal integrations: -- lower the complete workload through one frontend; -- call a whole-workload `search_workload*` function once; -- use the standard strategy factory rather than invoking individual strategies; -- preserve explicit accuracy targets and required evidence; -- use lifecycle-aware selection before claiming that a maintained summary is - preferable to exact recomputation; and -- treat physical compilation and deployment as a downstream step. - -The lower-level public traits and functions support research and deployment -extensions. They are not additional mandatory stages and should not be -presented as independent end-user workflows. - -Public Rust visibility does not by itself make a type part of the recommended -integration surface. New public enums, variants, and extension points require -a concrete workflow that consumes them. API review should remove or internalize -duplicate concepts, unused variants, and compatibility types after their -consumers have migrated. This document names the intended external concepts; -the library reference records the current Rust entry points. +1. Lower the complete workload through a frontend. +2. Call a whole-workload `search_workload*` API. +3. Use the standard strategy set unless extending Planner. +4. Preserve explicit accuracy requirements and required evidence. +5. Use lifecycle-aware selection before making deployment cost decisions. +6. Treat physical compilation and execution as downstream responsibilities. + +Lower-level Planner traits and APIs are extension points for research and deployment-specific customization; they are not separate required workflow stages. ## Related documents -- [Planner pipeline](../concepts/planner-pipeline.md) -- [Pre-ASAP IR](../concepts/pre-asap-ir.md) -- [Post-ASAP IR](../concepts/post-asap-ir.md) -- [Planner/downstream boundary](planner-downstream-boundary.md) -- [Plan search internals](asap-aware-plan-search.md) -- [Public library reference](../../develop_docs/library-api.md) +* [Planner pipeline](../concepts/planner-pipeline.md) +* [Pre-ASAP IR](../concepts/pre-asap-ir.md) +* [Post-ASAP IR](../concepts/post-asap-ir.md) +* [Planner/downstream boundary](planner-downstream-boundary.md) +* [Plan search internals](asap-aware-plan-search.md) +* [Public library reference](../../develop_docs/library-api.md) From f83963e59f309434bf089d4711f2e00bc40e6fb3 Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Fri, 18 Sep 2026 14:45:54 -0400 Subject: [PATCH 04/22] Update input-output-workflow.md --- docs/design_docs/architecture/input-output-workflow.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 209fc73e..70a7ae32 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -17,14 +17,6 @@ canonical queries + requirements + planning context (legal Post-ASAP alternatives) ``` -A typical integration: - -1. Lower source queries into canonical Pre-ASAP `QueryExpr` roots. -2. Attach query IDs and accuracy requirements. -3. Run Planner over the workload. -4. Inspect the resulting alternatives or select a Post-ASAP DAG. -5. Bind the selected logical DAG to physical operators downstream. - If required accuracy, semantic, capability, or cost evidence is unavailable, Planner does not assume it. Unsupported optimizations fail closed, while `KeepPreAsap` preserves exact computation where supported. --- From a045645ab31cc896db3f1500f27e2f3f8fb118e8 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 18:47:36 +0000 Subject: [PATCH 05/22] docs: describe planning workload fields --- .../architecture/input-output-workflow.md | 146 ++++++++++++++++-- 1 file changed, 134 insertions(+), 12 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 70a7ae32..1708f736 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -23,21 +23,143 @@ If required accuracy, semantic, capability, or cost evidence is unavailable, Pla ## Inputs -### Source-language inputs +### Planning workload -A frontend converts source-language queries into Planner's canonical representation. +A frontend receives one `PlanningWorkload`. Query demand and facts about the +queried data are separate because they have different sources and update +cycles: -| Input | Required | Purpose | -| ------------------------------- | ---------------------: | -------------------------------------------------------- | -| Query text | Yes | Defines the query to plan. | -| Query language | Yes | Selects the appropriate frontend. | -| Schema/function catalog | Frontend-dependent | Resolves names and types. | -| Accuracy requirement | Yes | Use `Exact` when approximation is not allowed. | -| Query recurrence/time selection | For lifecycle planning | Describes how often and over what period the query runs. | -| Data ingestion interval | For PromQL lowering | Defines the selection horizon for bare selectors. | -| Other workload facts | Optional | Enable optimizations that depend on them. | +```rust +struct PlanningWorkload { + query_workload: QueryWorkload, + data_workload: Option, +} +``` + +| Field | Required | Purpose | +|---|---:|---| +| `query_workload` | Yes | Contains the source language and every one-time or repeating query. | +| `data_workload` | Optional generally; required for PromQL | Describes data arrival and evidence about ingestion, cardinality, and distribution. PromQL additionally requires a nonzero `data_ingestion_interval`. | + +Frontend-specific resolution inputs are supplied alongside this structure. For +example, SQL lowering requires a schema/function catalog. They are frontend +dependencies, not fields of `PlanningWorkload`. + +### Query workload + +`QueryWorkload` describes query demand. It deliberately does not describe +whether source data is still arriving. + +```rust +struct QueryWorkload { + language: QueryLanguage, + query_batch: Option>, + repeating_queries: Option>, +} +``` + +| Field | Required | Purpose | +|---|---:|---| +| `language` | Yes | Source language shared by every entry: PromQL, a SQL dialect, DataFusion, or Elastic DSL. It selects the frontend. | +| `query_batch` | Optional | Finite one-time query entries. `None` means there is no batch portion. | +| `repeating_queries` | Optional | Recurrent query entries. `None` means there is no repeating portion. | + +Both entry collections may be present. `QueryWorkload::entries()` normalizes +them into one ordered stream: batch entries first, followed by repeating +entries. If both are absent, lowering produces no query roots. + +#### One-time query fields + +```rust +struct BatchEntry { + query: Query, + requirements: QueryRequirements, + predictability: Predictability, + invocations: u64, + execute_at: Option, + time_selection: TimeSelection, +} +``` + +| Field | Required | Purpose | +|---|---:|---| +| `query` | Yes | Raw query text in `QueryWorkload.language`. | +| `requirements` | Yes | Accuracy and response-latency requirements. Defaults mean exact accuracy and unspecified latency. | +| `predictability` | Yes | Whether the query is ad hoc, known in advance, or unknown. `known_at` may record when a predictable query became known. | +| `invocations` | Yes, nonzero | Number of executions in this finite batch. | +| `execute_at` | Optional | Known execution time. Absence prevents time-specific preparation decisions. | +| `time_selection` | Yes | Whether the query follows current data or a historical interval, its lookback, and any fixed upper bound. Unknown/default values limit lifecycle reasoning. | + +#### Repeating query fields + +```rust +struct RepeatingEntry { + query: Query, + demand: RepeatedDemand, + requirements: QueryRequirements, + predictability: Predictability, + time_selection: TimeSelection, +} +``` + +| Field | Required | Purpose | +|---|---:|---| +| `query` | Yes | Raw query text in `QueryWorkload.language`. | +| `demand` | Yes | A nonzero fixed interval, fixed interval with evaluation phase, nonempty explicit schedule, or evidence-backed estimated rate. | +| `requirements` | Yes | Accuracy and response-latency requirements. | +| `predictability` | Yes | Whether future executions are known in advance. This is independent of recurrence. | +| `time_selection` | Yes | Event-time scope, optional lookback, and optional fixed `as_of` time. | + +`QueryRequirements` and `TimeSelection` expand as follows: + +| Structure | Field | Meaning | +|---|---|---| +| `QueryRequirements` | `accuracy` | `Explicit(AccuracyTarget)` or `ImplicitExact`. Use an explicit target when approximation is allowed. | +| `QueryRequirements` | `response_latency` | An optional finite, non-negative maximum in milliseconds. The default is unspecified. | +| `TimeSelection` | `scope` | `RealTime`, `Longitudinal`, `Mixed`, or `Unknown`. | +| `TimeSelection` | `lookback` | Optional event-time duration selected before the upper bound. | +| `TimeSelection` | `as_of` | Optional fixed upper-bound timestamp; `None` means planning/evaluation time. | + +Frontend lowering produces one Pre-ASAP `QueryExpr` root for each normalized +query entry. The caller must retain each root's association with its workload +entry for later recurrence and lifecycle planning. + +### Data workload + +`DataWorkload` describes the data being queried. Each empirical field uses +`Evidence` so a value is accompanied by its source and freshness. + +```rust +struct DataWorkload { + arrival: DataArrival, + data_ingestion_interval: Evidence, + ingestion_volume: Evidence, + ingestion_rate: Evidence, + input_cardinality: Evidence, + distribution: Evidence, +} +``` -Frontend lowering produces one Pre-ASAP `QueryExpr` root for each workload query. +| Field | Required | Purpose and behavior when unavailable | +|---|---:|---| +| `arrival` | Present; may be `Unknown` | Distinguishes data at rest, continuously ingesting data, and mixed data. Continuous-maintenance decisions are limited when unknown. | +| `data_ingestion_interval` | Required and nonzero for PromQL; otherwise optional | Sampling cadence for each PromQL source. Bare instant selectors use it as their explicit selection horizon. | +| `ingestion_volume` | Optional evidence | Total ingestion volume when known. Dependent resource estimates remain unavailable when absent or stale. | +| `ingestion_rate` | Optional evidence | Updates per second used to price continuous maintenance. It must be finite and non-negative; at-rest data cannot declare a positive rate. | +| `input_cardinality` | Optional evidence | Input row/sample count used by applicable sizing, accuracy, or cost rules. | +| `distribution` | Optional evidence | `Zipf`, `Uniform`, or `Bursty` key distribution used only by rules that explicitly consume it. | + +Every `Evidence` has four fields: + +| Field | Meaning | +|---|---| +| `value: Option` | The fact itself; `None` means unavailable. | +| `source: EvidenceSource` | `Declared`, `Observed`, `Derived`, or `Unknown`. | +| `observed_at_ms: Option` | Observation timestamp used for freshness checks. | +| `valid_for_ms: Option` | Validity duration. A duration without an observation timestamp is unusable. | + +Unavailable or stale data evidence stays unknown. Planner does not reinterpret +it as zero ingestion, zero cardinality, or a favorable distribution. ### Canonical Planner inputs From 517353264486d5ed7898733c2c30be68a717cf5f Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Fri, 18 Sep 2026 14:48:59 -0400 Subject: [PATCH 06/22] Update input-output-workflow.md --- .../architecture/input-output-workflow.md | 21 +------------------ 1 file changed, 1 insertion(+), 20 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 1708f736..6fed9ded 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -1,25 +1,6 @@ # ASAPPlanner input, output, and workflows -## Overview - -ASAPPlanner is a **logical planning library**. It takes canonical queries and their requirements, explores legal exact and approximate implementations, and returns a logical plan space. - -It does **not** deploy or execute plans. - -```text -canonical queries + requirements + planning context - | - v - ASAPPlanner - | - v - PlanSpace - (legal Post-ASAP alternatives) -``` - -If required accuracy, semantic, capability, or cost evidence is unavailable, Planner does not assume it. Unsupported optimizations fail closed, while `KeepPreAsap` preserves exact computation where supported. - ---- + ## Inputs From 9e74b15179fa3d94269624a84fc6f3ae6c0fb2dc Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 18:56:21 +0000 Subject: [PATCH 07/22] docs: summarize planner input and output fields --- .../architecture/input-output-workflow.md | 58 ++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 6fed9ded..402bcd04 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -1,6 +1,62 @@ # ASAPPlanner input, output, and workflows - +## Overview + +ASAPPlanner is a **logical planning library**. Its input is a planning workload +plus the models, evidence, and deployment capabilities needed by the requested +planning workflow. Its canonical output is a `PlanSpace` containing the legal +Post-ASAP alternatives for the workload. + +### Input fields at a glance + +| Input | Fields | Required | +|---|---|---:| +| `PlanningWorkload.query_workload` | `language`, `query_batch`, `repeating_queries` | Yes | +| `BatchEntry` | `query`, `requirements`, `predictability`, `invocations`, `execute_at`, `time_selection` | For each one-time query | +| `RepeatingEntry` | `query`, `demand`, `requirements`, `predictability`, `time_selection` | For each repeating query | +| `PlanningWorkload.data_workload` | `arrival`, `data_ingestion_interval`, `ingestion_volume`, `ingestion_rate`, `input_cardinality`, `distribution` | Conditional: required for PromQL; individual facts are required only by workflows that consume them | +| Frontend context | Schema/function catalog and other language-specific resolution inputs | Frontend-dependent | +| Planning context | Accuracy target/model/evidence, cost model, planning time, horizon, and lifecycle capabilities | Workflow-dependent | + +Frontend lowering converts the workload entries into canonical Pre-ASAP +`QueryExpr` roots. The planning core associates each root with its caller-owned +query ID and accuracy target. + +### Output fields at a glance + +| Output | Fields or contents | Meaning | +|---|---|---| +| `PlanSpace` | Canonical workload roots, memo groups, legal candidates, rejected candidates and reasons, cross-group composition information | Canonical ASAPPlanner output: the complete logical choice space | +| `Vec` | `target`, `consumer_count`, index-aligned `candidates` and `costs` | Optional ranked view of the same `PlanSpace` | +| Materialized Post-ASAP roots | One `Rc` DAG per selected workload root, with shared nodes where applicable | Optional result after whole-plan logical selection | +| `SummaryMaintenanceLifecyclePlan` | `root`, lifecycle deployments, horizon/rates/expected reads, window realization, accuracy guarantee, summary/raw costs, raw-recompute decision | Optional lifecycle-aware selected result | + +These outputs are logical planning artifacts. ASAPPlanner does **not** produce a +deployed executable plan; downstream systems bind physical operators, choose +placement and storage, deploy state, and execute queries. + +```text +PlanningWorkload + frontend context + | + v + canonical QueryExpr roots + + + models + evidence + capabilities + | + v + ASAPPlanner + | + v + PlanSpace: legal Post-ASAP alternatives + | + +--> ranked view (optional) + +--> selected DAGs (optional) + +--> lifecycle plans (optional) +``` + +If required accuracy, semantic, capability, or cost evidence is unavailable, Planner does not assume it. Unsupported optimizations fail closed, while `KeepPreAsap` preserves exact computation where supported. + +--- ## Inputs From 8ecb6686fce1136794cd9cb1dc68a42dc6ac0d7a Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 18:58:53 +0000 Subject: [PATCH 08/22] docs: keep overview at workload boundary --- .../architecture/input-output-workflow.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 402bcd04..6cca5714 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -11,16 +11,14 @@ Post-ASAP alternatives for the workload. | Input | Fields | Required | |---|---|---:| -| `PlanningWorkload.query_workload` | `language`, `query_batch`, `repeating_queries` | Yes | -| `BatchEntry` | `query`, `requirements`, `predictability`, `invocations`, `execute_at`, `time_selection` | For each one-time query | -| `RepeatingEntry` | `query`, `demand`, `requirements`, `predictability`, `time_selection` | For each repeating query | -| `PlanningWorkload.data_workload` | `arrival`, `data_ingestion_interval`, `ingestion_volume`, `ingestion_rate`, `input_cardinality`, `distribution` | Conditional: required for PromQL; individual facts are required only by workflows that consume them | -| Frontend context | Schema/function catalog and other language-specific resolution inputs | Frontend-dependent | -| Planning context | Accuracy target/model/evidence, cost model, planning time, horizon, and lifecycle capabilities | Workflow-dependent | +| `PlanningWorkload.query_workload` | Query language and one-time/repeating query workloads | Yes | +| `PlanningWorkload.data_workload` | Data arrival and optional evidence about ingestion, cardinality, and distribution | Conditional: required for PromQL; otherwise optional | Frontend lowering converts the workload entries into canonical Pre-ASAP `QueryExpr` roots. The planning core associates each root with its caller-owned -query ID and accuracy target. +query ID and accuracy target. Frontend-specific context and planning models are +workflow parameters rather than fields of `PlanningWorkload`; later sections +document them separately. ### Output fields at a glance From 0a8c001c78ed128acf7536c1fccb6a8ff46bd0f6 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 19:04:03 +0000 Subject: [PATCH 09/22] docs: define one planner output --- .../architecture/input-output-workflow.md | 33 ++++++++++--------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 6cca5714..4f14dc44 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -20,18 +20,20 @@ query ID and accuracy target. Frontend-specific context and planning models are workflow parameters rather than fields of `PlanningWorkload`; later sections document them separately. -### Output fields at a glance +### Output at a glance | Output | Fields or contents | Meaning | |---|---|---| -| `PlanSpace` | Canonical workload roots, memo groups, legal candidates, rejected candidates and reasons, cross-group composition information | Canonical ASAPPlanner output: the complete logical choice space | -| `Vec` | `target`, `consumer_count`, index-aligned `candidates` and `costs` | Optional ranked view of the same `PlanSpace` | -| Materialized Post-ASAP roots | One `Rc` DAG per selected workload root, with shared nodes where applicable | Optional result after whole-plan logical selection | -| `SummaryMaintenanceLifecyclePlan` | `root`, lifecycle deployments, horizon/rates/expected reads, window realization, accuracy guarantee, summary/raw costs, raw-recompute decision | Optional lifecycle-aware selected result | +| `PlanSpace` | The legal candidate Post-ASAP DAGs for the workload, represented compactly as canonical roots, memoized alternatives, and cross-group composition information | The single ASAPPlanner output | -These outputs are logical planning artifacts. ASAPPlanner does **not** produce a -deployed executable plan; downstream systems bind physical operators, choose -placement and storage, deploy state, and execute queries. +`PlanSpace` is a compact representation of a set of candidate DAGs rather than +an eagerly enumerated `Vec`. Ranking, selection, materialization, and +lifecycle APIs operate on this candidate set; they are views or helper +operations, not additional top-level Planner outputs. + +The candidate DAGs are logical planning artifacts. ASAPPlanner does **not** +produce a deployed executable plan; downstream systems bind physical operators, +choose placement and storage, deploy state, and execute queries. ```text PlanningWorkload + frontend context @@ -46,10 +48,6 @@ PlanningWorkload + frontend context | v PlanSpace: legal Post-ASAP alternatives - | - +--> ranked view (optional) - +--> selected DAGs (optional) - +--> lifecycle plans (optional) ``` If required accuracy, semantic, capability, or cost evidence is unavailable, Planner does not assume it. Unsupported optimizations fail closed, while `KeepPreAsap` preserves exact computation where supported. @@ -253,7 +251,7 @@ When required evidence is missing, the dependent optimization is unavailable. --- -## Outputs +## Output ### `PlanSpace` @@ -267,6 +265,11 @@ When required evidence is missing, the dependent optimization is unavailable. A `PlanSpace` represents a **space of logical DAG choices**, not a single executable plan. +The remaining APIs in this section derive information from that one output; +they do not define separate ASAPPlanner output contracts. + +### Ranked view + `PlanSpace::cost_sorted` provides a ranked view for inspection: ```text @@ -280,7 +283,7 @@ Vec Date: Fri, 18 Sep 2026 19:05:20 +0000 Subject: [PATCH 10/22] docs: name frontend-specific dependencies --- .../architecture/input-output-workflow.md | 23 +++++++++++++------ 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 4f14dc44..a3d36c76 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -16,9 +16,11 @@ Post-ASAP alternatives for the workload. Frontend lowering converts the workload entries into canonical Pre-ASAP `QueryExpr` roots. The planning core associates each root with its caller-owned -query ID and accuracy target. Frontend-specific context and planning models are -workflow parameters rather than fields of `PlanningWorkload`; later sections -document them separately. +query ID and accuracy target. Some frontends take additional explicit +dependencies that are not fields of `PlanningWorkload`: PromQL takes the +planning timestamp and optionally a histogram catalog; SQL takes a schema and +function catalog; MetricsQL takes no additional catalog. Planning models are +separate inputs to candidate search, not frontend dependencies. ### Output at a glance @@ -36,7 +38,7 @@ produce a deployed executable plan; downstream systems bind physical operators, choose placement and storage, deploy state, and execute queries. ```text -PlanningWorkload + frontend context +PlanningWorkload + frontend-specific dependencies | v canonical QueryExpr roots @@ -74,9 +76,16 @@ struct PlanningWorkload { | `query_workload` | Yes | Contains the source language and every one-time or repeating query. | | `data_workload` | Optional generally; required for PromQL | Describes data arrival and evidence about ingestion, cardinality, and distribution. PromQL additionally requires a nonzero `data_ingestion_interval`. | -Frontend-specific resolution inputs are supplied alongside this structure. For -example, SQL lowering requires a schema/function catalog. They are frontend -dependencies, not fields of `PlanningWorkload`. +Frontend-specific dependencies are supplied alongside this structure: + +| Frontend | Additional lowering inputs | +|---|---| +| PromQL | `now_ms`, plus an optional `HistogramCatalog` when histogram semantics must be resolved | +| SQL | `SqlCatalog`; single-query APIs also receive the accuracy target and optionally an explicit SQL dialect | +| MetricsQL | No catalog; the single-query API receives the query text and accuracy target directly | + +These are explicit frontend function arguments, not one generic +`FrontendContext` type and not fields of `PlanningWorkload`. ### Query workload From ddedb2adf6fa2de8a2ca2fe9b940763f6033338e Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 19:07:20 +0000 Subject: [PATCH 11/22] docs: explain compact plan space representation --- .../architecture/input-output-workflow.md | 21 +++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index a3d36c76..ea21d465 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -28,10 +28,23 @@ separate inputs to candidate search, not frontend dependencies. |---|---|---| | `PlanSpace` | The legal candidate Post-ASAP DAGs for the workload, represented compactly as canonical roots, memoized alternatives, and cross-group composition information | The single ASAPPlanner output | -`PlanSpace` is a compact representation of a set of candidate DAGs rather than -an eagerly enumerated `Vec`. Ranking, selection, materialization, and -lifecycle APIs operate on this candidate set; they are views or helper -operations, not additional top-level Planner outputs. +`PlanSpace` does not eagerly copy every complete DAG. It stores the workload's +canonical roots once, creates one memo group for each distinct target sub-DAG, +and stores that target's replacement alternatives once inside the group. +Candidate children refer back to canonical targets, so common subexpressions +and shared alternatives are not duplicated across roots. + +For example, if one target has three alternatives and its child has two, +eager enumeration could create six complete DAGs. `PlanSpace` stores the three +parent alternatives, the two child alternatives, and their relationship. +Whole-plan selection chooses compatible alternatives across those groups; +materialization then recursively substitutes the selected alternatives to +construct a complete Post-ASAP DAG. This memoized representation avoids the +Cartesian-product expansion of complete DAGs and preserves shared nodes. + +Ranking, selection, materialization, and lifecycle APIs operate on this same +candidate set; they are views or helper operations, not additional top-level +Planner outputs. The candidate DAGs are logical planning artifacts. ASAPPlanner does **not** produce a deployed executable plan; downstream systems bind physical operators, From 449c56f1a2326e6c71af4f7876666e9484ad8f8d Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 19:08:37 +0000 Subject: [PATCH 12/22] docs: move plan space details after overview --- .../architecture/input-output-workflow.md | 34 +++++++++---------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index ea21d465..daba19f9 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -28,23 +28,8 @@ separate inputs to candidate search, not frontend dependencies. |---|---|---| | `PlanSpace` | The legal candidate Post-ASAP DAGs for the workload, represented compactly as canonical roots, memoized alternatives, and cross-group composition information | The single ASAPPlanner output | -`PlanSpace` does not eagerly copy every complete DAG. It stores the workload's -canonical roots once, creates one memo group for each distinct target sub-DAG, -and stores that target's replacement alternatives once inside the group. -Candidate children refer back to canonical targets, so common subexpressions -and shared alternatives are not duplicated across roots. - -For example, if one target has three alternatives and its child has two, -eager enumeration could create six complete DAGs. `PlanSpace` stores the three -parent alternatives, the two child alternatives, and their relationship. -Whole-plan selection chooses compatible alternatives across those groups; -materialization then recursively substitutes the selected alternatives to -construct a complete Post-ASAP DAG. This memoized representation avoids the -Cartesian-product expansion of complete DAGs and preserves shared nodes. - -Ranking, selection, materialization, and lifecycle APIs operate on this same -candidate set; they are views or helper operations, not additional top-level -Planner outputs. +Ranking, selection, materialization, and lifecycle APIs are views or helper +operations over this output, not additional top-level Planner outputs. The candidate DAGs are logical planning artifacts. ASAPPlanner does **not** produce a deployed executable plan; downstream systems bind physical operators, @@ -287,6 +272,21 @@ When required evidence is missing, the dependent optimization is unavailable. A `PlanSpace` represents a **space of logical DAG choices**, not a single executable plan. +It represents that space compactly instead of eagerly copying every complete +DAG. `PlanSpace` stores the workload's canonical roots once, creates one memo +group for each distinct target sub-DAG, and stores that target's replacement +alternatives once inside the group. Candidate children refer back to canonical +targets, so common subexpressions and shared alternatives are not duplicated +across roots. + +For example, if one target has three alternatives and its child has two, +eager enumeration could create six complete DAGs. `PlanSpace` stores the three +parent alternatives, the two child alternatives, and their relationship. +Whole-plan selection chooses compatible alternatives across those groups; +materialization then recursively substitutes the selected alternatives to +construct a complete Post-ASAP DAG. This memoized representation avoids the +Cartesian-product expansion of complete DAGs and preserves shared nodes. + The remaining APIs in this section derive information from that one output; they do not define separate ASAPPlanner output contracts. From 80b48cfb46d4f8e7e883290cc6af618be074adf2 Mon Sep 17 00:00:00 2001 From: Zeying Zhu <50204836+zzylol@users.noreply.github.com> Date: Fri, 18 Sep 2026 15:09:46 -0400 Subject: [PATCH 13/22] Update input-output-workflow.md --- docs/design_docs/architecture/input-output-workflow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index daba19f9..45d34dcb 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -26,7 +26,7 @@ separate inputs to candidate search, not frontend dependencies. | Output | Fields or contents | Meaning | |---|---|---| -| `PlanSpace` | The legal candidate Post-ASAP DAGs for the workload, represented compactly as canonical roots, memoized alternatives, and cross-group composition information | The single ASAPPlanner output | +| `PlanSpace` | The legal candidate Post-ASAP DAGs for the workload, represented compactly as canonical roots, memoized alternatives, and cross-group composition information | The ASAPPlanner output | Ranking, selection, materialization, and lifecycle APIs are views or helper operations over this output, not additional top-level Planner outputs. From 99a64e36bd5deeadd57a5c17d152c39c159daed8 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 19:09:51 +0000 Subject: [PATCH 14/22] docs: mirror workload nesting in input sections --- .../architecture/input-output-workflow.md | 46 +++++++++++-------- 1 file changed, 28 insertions(+), 18 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 45d34dcb..5211d745 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -56,7 +56,7 @@ If required accuracy, semantic, capability, or cost evidence is unavailable, Pla ## Inputs -### Planning workload +### `PlanningWorkload` A frontend receives one `PlanningWorkload`. Query demand and facts about the queried data are separate because they have different sources and update @@ -74,18 +74,7 @@ struct PlanningWorkload { | `query_workload` | Yes | Contains the source language and every one-time or repeating query. | | `data_workload` | Optional generally; required for PromQL | Describes data arrival and evidence about ingestion, cardinality, and distribution. PromQL additionally requires a nonzero `data_ingestion_interval`. | -Frontend-specific dependencies are supplied alongside this structure: - -| Frontend | Additional lowering inputs | -|---|---| -| PromQL | `now_ms`, plus an optional `HistogramCatalog` when histogram semantics must be resolved | -| SQL | `SqlCatalog`; single-query APIs also receive the accuracy target and optionally an explicit SQL dialect | -| MetricsQL | No catalog; the single-query API receives the query text and accuracy target directly | - -These are explicit frontend function arguments, not one generic -`FrontendContext` type and not fields of `PlanningWorkload`. - -### Query workload +#### `query_workload: QueryWorkload` `QueryWorkload` describes query demand. It deliberately does not describe whether source data is still arriving. @@ -108,7 +97,7 @@ Both entry collections may be present. `QueryWorkload::entries()` normalizes them into one ordered stream: batch entries first, followed by repeating entries. If both are absent, lowering produces no query roots. -#### One-time query fields +##### `query_batch: Option>` ```rust struct BatchEntry { @@ -130,7 +119,7 @@ struct BatchEntry { | `execute_at` | Optional | Known execution time. Absence prevents time-specific preparation decisions. | | `time_selection` | Yes | Whether the query follows current data or a historical interval, its lookback, and any fixed upper bound. Unknown/default values limit lifecycle reasoning. | -#### Repeating query fields +##### `repeating_queries: Option>` ```rust struct RepeatingEntry { @@ -150,7 +139,11 @@ struct RepeatingEntry { | `predictability` | Yes | Whether future executions are known in advance. This is independent of recurrence. | | `time_selection` | Yes | Event-time scope, optional lookback, and optional fixed `as_of` time. | -`QueryRequirements` and `TimeSelection` expand as follows: +##### Shared entry fields + +`BatchEntry` and `RepeatingEntry` both contain `QueryRequirements`, +`Predictability`, and `TimeSelection`. The nested requirement and time-selection +fields expand as follows: | Structure | Field | Meaning | |---|---|---| @@ -164,7 +157,7 @@ Frontend lowering produces one Pre-ASAP `QueryExpr` root for each normalized query entry. The caller must retain each root's association with its workload entry for later recurrence and lifecycle planning. -### Data workload +#### `data_workload: Option` `DataWorkload` describes the data being queried. Each empirical field uses `Evidence` so a value is accompanied by its source and freshness. @@ -189,7 +182,10 @@ struct DataWorkload { | `input_cardinality` | Optional evidence | Input row/sample count used by applicable sizing, accuracy, or cost rules. | | `distribution` | Optional evidence | `Zipf`, `Uniform`, or `Bursty` key distribution used only by rules that explicitly consume it. | -Every `Evidence` has four fields: +##### `Evidence` fields + +Every empirical field in `DataWorkload` uses `Evidence`, which has four +fields: | Field | Meaning | |---|---| @@ -201,6 +197,20 @@ Every `Evidence` has four fields: Unavailable or stale data evidence stays unknown. Planner does not reinterpret it as zero ingestion, zero cardinality, or a favorable distribution. +### Frontend-specific dependencies + +These inputs are supplied alongside `PlanningWorkload`, rather than nested +inside it: + +| Frontend | Additional lowering inputs | +|---|---| +| PromQL | `now_ms`, plus an optional `HistogramCatalog` when histogram semantics must be resolved | +| SQL | `SqlCatalog`; single-query APIs also receive the accuracy target and optionally an explicit SQL dialect | +| MetricsQL | No catalog; the single-query API receives the query text and accuracy target directly | + +They are explicit frontend function arguments, not one generic +`FrontendContext` type. + ### Canonical Planner inputs The planning core operates on: From f7c50b7ac28bbf379e0afbf3156e8d0c868e758b Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 19:10:34 +0000 Subject: [PATCH 15/22] docs: link overview to output helpers --- docs/design_docs/architecture/input-output-workflow.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 5211d745..3268bf7d 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -28,8 +28,10 @@ separate inputs to candidate search, not frontend dependencies. |---|---|---| | `PlanSpace` | The legal candidate Post-ASAP DAGs for the workload, represented compactly as canonical roots, memoized alternatives, and cross-group composition information | The ASAPPlanner output | -Ranking, selection, materialization, and lifecycle APIs are views or helper -operations over this output, not additional top-level Planner outputs. +[Ranking](#ranked-view), [selection and +materialization](#selection-and-materialization-helper), and +[lifecycle](#lifecycle-aware-helper) APIs are views or helper operations over +this output, not additional top-level Planner outputs. The candidate DAGs are logical planning artifacts. ASAPPlanner does **not** produce a deployed executable plan; downstream systems bind physical operators, From 4983f5b96e1733359c3e6315212375cc41137b01 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 19:13:24 +0000 Subject: [PATCH 16/22] docs: explain caller-owned plan root IDs --- .../architecture/input-output-workflow.md | 40 ++++++++++++------- 1 file changed, 26 insertions(+), 14 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 3268bf7d..2bd01068 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -213,25 +213,37 @@ inside it: They are explicit frontend function arguments, not one generic `FrontendContext` type. -### Canonical Planner inputs +### Candidate-search API inputs -The planning core operates on: +After frontend lowering, the configurable whole-workload search API receives: -```text -Vec<(query_id, Rc, Option)> +```rust +search_workload_with_targets( + roots: Vec<(Id, Rc, Option)>, + strategies: &[Box], + accuracy_model: &dyn AccuracyModel, +) ``` -The main inputs are: +`Id` is not a field of `PlanningWorkload`, and there is no Planner-defined +`query_id` type. Frontend lowering returns `QueryExpr` roots in normalized +workload-entry order. The integration layer pairs each root with an opaque `Id` +before search—for example, the workload entry index or an application query +identifier. Planner preserves that value so the caller can associate output +roots with its own queries; it does not affect candidate semantics or ranking. -* **`query_id`** — caller-owned query identity. -* **`QueryExpr`** — canonical exact query semantics. -* **`AccuracyTarget`** — required end-to-end accuracy, if enforced. -* **Strategies** — candidate transformations Planner may explore. -* **`AccuracyModel`** — determines how guarantees compose and satisfy targets. -* **`CostModel`** — evaluates candidate cost and availability. -* **`AccuracyEvidenceProvider`** — provides facts required to prove candidate guarantees. - -Most integrations should use the standard strategy set and `DefaultAccuracyModel`. +| API input | Source | Planning role | +|---|---|---| +| `Id` | Added by the integration layer after lowering | Opaque result correlation only | +| `Rc` | Frontend output | Canonical exact query semantics | +| `Option` | The corresponding workload entry's `requirements.accuracy` | Enforces an end-to-end root guarantee when present | +| `strategies` | Normally a standard strategy factory | Defines which candidate transformations search may explore | +| `accuracy_model` | Normally `DefaultAccuracyModel` | Composes guarantees and checks them against targets | + +`AccuracyEvidenceProvider` is supplied when constructing evidence-aware +strategies; it is not another field in the root tuple. `CostModel` is used by +strategy construction and by later ranking or selection helpers; it is not a +direct argument of `search_workload_with_targets`. ### Additional inputs for lifecycle planning From eba117465042758377b4b4c6e11c04c13e9d8d81 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 19:17:11 +0000 Subject: [PATCH 17/22] docs: keep inputs at end-to-end boundary --- .../architecture/input-output-workflow.md | 64 ++++--------------- 1 file changed, 13 insertions(+), 51 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 2bd01068..9a3ef7e7 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -14,13 +14,12 @@ Post-ASAP alternatives for the workload. | `PlanningWorkload.query_workload` | Query language and one-time/repeating query workloads | Yes | | `PlanningWorkload.data_workload` | Data arrival and optional evidence about ingestion, cardinality, and distribution | Conditional: required for PromQL; otherwise optional | -Frontend lowering converts the workload entries into canonical Pre-ASAP -`QueryExpr` roots. The planning core associates each root with its caller-owned -query ID and accuracy target. Some frontends take additional explicit -dependencies that are not fields of `PlanningWorkload`: PromQL takes the -planning timestamp and optionally a histogram catalog; SQL takes a schema and -function catalog; MetricsQL takes no additional catalog. Planning models are -separate inputs to candidate search, not frontend dependencies. +As part of the planning workflow, frontend lowering converts the workload +entries into canonical Pre-ASAP `QueryExpr` roots. Those roots and the +candidate-search API that consumes them are internal stages, not additional +end-to-end user inputs. Some frontends require explicit dependencies alongside +the workload; these are listed under +[Frontend-specific dependencies](#frontend-specific-dependencies). ### Output at a glance @@ -38,18 +37,13 @@ produce a deployed executable plan; downstream systems bind physical operators, choose placement and storage, deploy state, and execute queries. ```text -PlanningWorkload + frontend-specific dependencies - | - v - canonical QueryExpr roots - + - models + evidence + capabilities - | - v - ASAPPlanner - | - v - PlanSpace: legal Post-ASAP alternatives +PlanningWorkload + required workflow context + | + v + ASAPPlanner + | + v + PlanSpace: candidate Post-ASAP DAGs ``` If required accuracy, semantic, capability, or cost evidence is unavailable, Planner does not assume it. Unsupported optimizations fail closed, while `KeepPreAsap` preserves exact computation where supported. @@ -213,38 +207,6 @@ inside it: They are explicit frontend function arguments, not one generic `FrontendContext` type. -### Candidate-search API inputs - -After frontend lowering, the configurable whole-workload search API receives: - -```rust -search_workload_with_targets( - roots: Vec<(Id, Rc, Option)>, - strategies: &[Box], - accuracy_model: &dyn AccuracyModel, -) -``` - -`Id` is not a field of `PlanningWorkload`, and there is no Planner-defined -`query_id` type. Frontend lowering returns `QueryExpr` roots in normalized -workload-entry order. The integration layer pairs each root with an opaque `Id` -before search—for example, the workload entry index or an application query -identifier. Planner preserves that value so the caller can associate output -roots with its own queries; it does not affect candidate semantics or ranking. - -| API input | Source | Planning role | -|---|---|---| -| `Id` | Added by the integration layer after lowering | Opaque result correlation only | -| `Rc` | Frontend output | Canonical exact query semantics | -| `Option` | The corresponding workload entry's `requirements.accuracy` | Enforces an end-to-end root guarantee when present | -| `strategies` | Normally a standard strategy factory | Defines which candidate transformations search may explore | -| `accuracy_model` | Normally `DefaultAccuracyModel` | Composes guarantees and checks them against targets | - -`AccuracyEvidenceProvider` is supplied when constructing evidence-aware -strategies; it is not another field in the root tuple. `CostModel` is used by -strategy construction and by later ranking or selection helpers; it is not a -direct argument of `search_workload_with_targets`. - ### Additional inputs for lifecycle planning Lifecycle-aware planning compares maintaining a summary against recomputing the raw query. From afb9a68ec411a8f9232af576ec088c68525056c0 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 19:18:33 +0000 Subject: [PATCH 18/22] docs: move lifecycle parameters to helper --- .../architecture/input-output-workflow.md | 37 ++++++++----------- 1 file changed, 15 insertions(+), 22 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 9a3ef7e7..2929d380 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -37,13 +37,13 @@ produce a deployed executable plan; downstream systems bind physical operators, choose placement and storage, deploy state, and execute queries. ```text -PlanningWorkload + required workflow context - | - v - ASAPPlanner - | - v - PlanSpace: candidate Post-ASAP DAGs +PlanningWorkload + frontend-specific dependencies + | + v + ASAPPlanner + | + v + PlanSpace: candidate Post-ASAP DAGs ``` If required accuracy, semantic, capability, or cost evidence is unavailable, Planner does not assume it. Unsupported optimizations fail closed, while `KeepPreAsap` preserves exact computation where supported. @@ -207,21 +207,6 @@ inside it: They are explicit frontend function arguments, not one generic `FrontendContext` type. -### Additional inputs for lifecycle planning - -Lifecycle-aware planning compares maintaining a summary against recomputing the raw query. - -It additionally requires: - -* query workload bindings; -* planning time and horizon; -* query recurrence; -* data arrival/update rate; -* deployment lifecycle capabilities; and -* comparable cost information for summary and raw execution. - -Missing required information remains **unknown** rather than being treated as zero. - --- ## Evidence @@ -314,6 +299,14 @@ It is still **not an executable deployment plan**. Physical operator binding, pl ### Lifecycle-aware helper +Lifecycle-aware planning is a helper over the candidate space that compares +maintaining a summary against recomputing the raw query. Calling it requires +query-workload bindings, planning time and horizon, recurrence, data arrival +and update rate, deployment lifecycle capabilities, and comparable summary and +raw-execution cost information. These are helper parameters, not fields added +to the canonical `PlanningWorkload` input. Missing required information remains +unknown rather than being treated as zero. + `SummaryMaintenanceLifecyclePlan` additionally records: * the materialized root; From f11d3bf5c2c91e8505d78c1b0d3d6180d8225ad9 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 19:19:56 +0000 Subject: [PATCH 19/22] docs: classify evidence as planner input --- .../architecture/input-output-workflow.md | 25 +++++++++++-------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 2929d380..b96f2c77 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -13,6 +13,7 @@ Post-ASAP alternatives for the workload. |---|---|---:| | `PlanningWorkload.query_workload` | Query language and one-time/repeating query workloads | Yes | | `PlanningWorkload.data_workload` | Data arrival and optional evidence about ingestion, cardinality, and distribution | Conditional: required for PromQL; otherwise optional | +| Planning evidence and capabilities | Domain, accuracy, cost, and deployment facts supplied through the applicable provider/model interface | Conditional: required only by optimizations that depend on those facts | As part of the planning workflow, frontend lowering converts the workload entries into canonical Pre-ASAP `QueryExpr` roots. Those roots and the @@ -207,25 +208,27 @@ inside it: They are explicit frontend function arguments, not one generic `FrontendContext` type. ---- - -## Evidence +### Planning evidence inputs **Evidence is a scoped fact used to establish legality, accuracy, cost, or feasibility.** -Examples include: +Evidence is input to planning. The current library does not collect every kind +in one `PlanningWorkload` field; each fact enters through the interface that +consumes it: -| Evidence | Examples | -| --------------- | ---------------------------------------------------------------- | -| Semantic/domain | Input range, nonempty population, nonzero denominator | -| Accuracy | Quantile domain, Top-K confidence, composition certificate | -| Cost | CPU time, operation count, memory, scan/storage I/O | -| Workload | Cardinality, distribution, ingestion rate, recurrence | -| Capability | Supported summaries, merge/delete support, window implementation | +| Evidence | Examples | Supplied through | +|---|---|---| +| Semantic/domain | Input range, nonempty population, nonzero denominator | Typed accuracy/domain evidence provider | +| Accuracy | Quantile domain, Top-K confidence, composition certificate | Accuracy evidence provider or registered accuracy model | +| Cost | CPU time, operation count, memory, scan/storage I/O | Cost model or physical-evidence provider | +| Workload | Cardinality, distribution, ingestion rate, recurrence | `PlanningWorkload` query/data workload fields | +| Capability | Supported summaries, merge/delete support, window realization | Deployment capability or lifecycle provider | Evidence must apply to the relevant workload and implementation. Time-sensitive evidence should also carry freshness information. When required evidence is missing, the dependent optimization is unavailable. +Planner output may record the resulting guarantee, evidence provenance, or a +rejection reason, but evidence itself remains an input. --- From 3995bef5e8355968a34c92178d42f0fbb5bed702 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 19:21:03 +0000 Subject: [PATCH 20/22] docs: mark replanning as future work --- .../architecture/input-output-workflow.md | 28 ++++++------------- 1 file changed, 9 insertions(+), 19 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index b96f2c77..20fb1796 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -369,11 +369,13 @@ It is the recommended workflow for deployment decisions. --- -## Replanning +## Replanning (future support) -Replanning uses the same interface as initial planning. +> **Status: Future support.** ASAPPlanner does not currently define an +> end-to-end replanning or deployment-transition contract. -Run Planner again whenever a selection-relevant input changes, such as: +The intended design will reuse the initial-planning input boundary. A caller +would request replanning whenever a selection-relevant input changes, such as: * query semantics or accuracy requirements; * recurrence, horizon, data arrival, or distribution; @@ -395,22 +397,10 @@ updated workload + evidence + capabilities downstream deployment transition ``` -Planner produces a new logical decision. The downstream system owns deployment diffing, migration, activation, and rollback. - ---- - -## Integration guidance - -For normal integrations: - -1. Lower the complete workload through a frontend. -2. Call a whole-workload `search_workload*` API. -3. Use the standard strategy set unless extending Planner. -4. Preserve explicit accuracy requirements and required evidence. -5. Use lifecycle-aware selection before making deployment cost decisions. -6. Treat physical compilation and execution as downstream responsibilities. - -Lower-level Planner traits and APIs are extension points for research and deployment-specific customization; they are not separate required workflow stages. +Today, a caller can run Planner again and obtain a new logical candidate space, +but ASAPPlanner does not relate that result to the previous plan. Future support +must define plan identity and compatibility across runs. Deployment diffing, +migration, activation, and rollback remain downstream responsibilities. ## Related documents From 03c712e034ff169d1f474794caff8a2960c6a5b8 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 19:22:18 +0000 Subject: [PATCH 21/22] docs: state when planning evidence is required --- .../architecture/input-output-workflow.md | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index 20fb1796..bbdfa51a 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -216,17 +216,22 @@ Evidence is input to planning. The current library does not collect every kind in one `PlanningWorkload` field; each fact enters through the interface that consumes it: -| Evidence | Examples | Supplied through | -|---|---|---| -| Semantic/domain | Input range, nonempty population, nonzero denominator | Typed accuracy/domain evidence provider | -| Accuracy | Quantile domain, Top-K confidence, composition certificate | Accuracy evidence provider or registered accuracy model | -| Cost | CPU time, operation count, memory, scan/storage I/O | Cost model or physical-evidence provider | -| Workload | Cardinality, distribution, ingestion rate, recurrence | `PlanningWorkload` query/data workload fields | -| Capability | Supported summaries, merge/delete support, window realization | Deployment capability or lifecycle provider | +Evidence is not globally required or globally optional. Each item is +**conditionally required** by the decision that consumes it: + +| Evidence | Required when | Examples | Supplied through | +|---|---|---|---| +| Semantic/domain | A transformation needs to prove an input precondition | Input range, nonempty population, nonzero denominator | Typed accuracy/domain evidence provider | +| Accuracy | An approximate candidate needs a data-dependent accuracy certificate | Quantile domain, Top-K confidence, composition certificate | Accuracy evidence provider or registered accuracy model | +| Cost | Candidates are compared or selected by deployment cost | CPU time, operation count, memory, scan/storage I/O | Cost model or physical-evidence provider | +| Workload | A frontend or optimization consumes that workload fact | Cardinality, distribution, ingestion rate, recurrence | `PlanningWorkload` query/data workload fields | +| Capability | A candidate must be checked against deployable operations | Supported summaries, merge/delete support, window realization | Deployment capability or lifecycle provider | Evidence must apply to the relevant workload and implementation. Time-sensitive evidence should also carry freshness information. -When required evidence is missing, the dependent optimization is unavailable. +Missing optional evidence does not invalidate unrelated candidates. When a +candidate requires missing evidence, that candidate is unavailable rather than +planned using a favorable assumption. Planner output may record the resulting guarantee, evidence provenance, or a rejection reason, but evidence itself remains an input. From 235eb6a908a4ff76a0602e897aa3195edf65340b Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 19:33:54 +0000 Subject: [PATCH 22/22] docs: separate lifecycle helper parameters --- .../architecture/input-output-workflow.md | 45 +++++++++++++------ 1 file changed, 32 insertions(+), 13 deletions(-) diff --git a/docs/design_docs/architecture/input-output-workflow.md b/docs/design_docs/architecture/input-output-workflow.md index bbdfa51a..c7e059b4 100644 --- a/docs/design_docs/architecture/input-output-workflow.md +++ b/docs/design_docs/architecture/input-output-workflow.md @@ -235,6 +235,12 @@ planned using a favorable assumption. Planner output may record the resulting guarantee, evidence provenance, or a rejection reason, but evidence itself remains an input. +This completes the canonical input boundary for producing `PlanSpace`. +Lifecycle-specific values such as a planning horizon and deployment lifecycle +capabilities are not additional `PlanSpace` inputs. They are parameters to the +optional [lifecycle-aware helper](#lifecycle-aware-helper) described after the +output. + --- ## Output @@ -307,13 +313,26 @@ It is still **not an executable deployment plan**. Physical operator binding, pl ### Lifecycle-aware helper -Lifecycle-aware planning is a helper over the candidate space that compares -maintaining a summary against recomputing the raw query. Calling it requires -query-workload bindings, planning time and horizon, recurrence, data arrival -and update rate, deployment lifecycle capabilities, and comparable summary and -raw-execution cost information. These are helper parameters, not fields added -to the canonical `PlanningWorkload` input. Missing required information remains -unknown rather than being treated as zero. +Lifecycle-aware planning is an optional operation on an existing `PlanSpace`. +It is not part of the canonical input-to-`PlanSpace` operation. Its purpose is +to compare maintaining a summary with recomputing the raw query. + +The public helper receives these parameters: + +| Helper parameter | Source | Required | +|---|---|---:| +| `PlanSpace` | Canonical ASAPPlanner output | Yes | +| Workload binding | `QueryWorkload` plus the workload-entry indices associated with each root | Yes | +| Planning time (`now_ms`) | Caller clock in Unix milliseconds | Yes | +| Planning horizon | Caller policy | Conditional: required for finite totals over recurring demand | +| Data arrival and update rate | `DataWorkload` evidence | Conditional: required to cost continuous maintenance | +| Lifecycle capabilities | Deployment/runtime provider | Yes for checking deployable lifecycle alternatives | +| Summary and raw cost information | Cost model and physical-evidence provider | Yes for a cost-based maintenance-versus-recompute decision | + +Recurrence and time selection are already fields of the bound `QueryWorkload`; +they are not duplicated as separate top-level inputs. Similarly, data arrival +and update rate are read from the optional `DataWorkload`. Missing required +facts remain unknown rather than being treated as zero. `SummaryMaintenanceLifecyclePlan` additionally records: @@ -360,15 +379,15 @@ This produces structurally compatible logical plans. It does not determine wheth Use when deciding whether maintained summary state should actually be deployed. ```text -PlanningWorkload + Pre-ASAP roots - -> search_workload_with_targets - -> global_selection_with_summary_maintenance_lifecycles - -> materialize_with_summary_maintenance_lifecycles - -> lifecycle-aware Post-ASAP plans +PlanSpace + lifecycle helper parameters + -> lifecycle-aware selection and materialization + -> SummaryMaintenanceLifecyclePlan -> downstream physical deployment ``` -This workflow considers recurrence, data arrival, planning horizon, capabilities, and comparable summary/raw costs. +This helper considers recurrence, data arrival, planning horizon, capabilities, +and comparable summary/raw costs. It does not change the canonical +`PlanningWorkload -> PlanSpace` interface. It is the recommended workflow for deployment decisions.