Skip to content
Merged
256 changes: 13 additions & 243 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,250 +1,20 @@
# ASAPPlanner — design
# ASAPPlanner

ASAPPlanner is a system to map a query workload to an ASAP plan. A query workload is a batch of queries or a set of repeating queries, in any query language like PromQL or SQL. An ASAP plan is a query plan (like in databases) that uses ASAP primitives like sketches, exact summaries, wavelets, etc.
ASAPPlanner turns SQL, PromQL, and MetricsQL query workloads into legal candidate plans that may use Approximate Streaming Analytics Primitives (ASAPs), such as sketches and exact summaries. It normalizes language-specific queries into a shared representation, then enumerates and ranks semantically equivalent alternatives. Downstream systems choose, deploy, and execute a physical plan.

ASAPPlanner does this mapping in 2 steps:
(1) normalizing query workloads from different query languages into a common intermediate representation (IR)
(2) mapping the IR to an ASAP plan
## Start here

Separating these 2 steps is helpful to decouple concerns and for extensibility.
Step 1 **interprets** the query workload semantics and **normalizes** them into our own IR. Adding a new query language or dialect can be done by extending step 1 and not touching step 2.
Step 2 **maps** query workload semantics to ASAP primitives. Adding a new ASAP primitive can be done by extending step 2 and not touching step 1.
- New to the repository? Read the [planner pipeline](docs/design_docs/concepts/planner-pipeline.md), then the [glossary](docs/design_docs/concepts/glossary.md).
- Want to run a query? Follow [Run and inspect a query](docs/user_guide_docs/run-a-query.md).
- Embedding Planner? Use the [library API guide](docs/develop_docs/library-api.md).
- Extending Planner? Start with the [ASAP-aware mapping architecture](docs/develop_docs/asap-aware-mapping-architecture.md), then [extend ASAP-aware mapping](docs/develop_docs/extend-asap-aware-mapping.md).
- Evaluating a design? Browse the [design documentation](docs/design_docs/README.md), [developer documentation](docs/develop_docs/README.md), and [user guides](docs/user_guide_docs/README.md).

1. **Interpretation** — understand a language-specific query and construct semantic intent.
2. **Intent canonicalization** — normalize equivalent queries from different languages into a common IR
3. **Mapping intents to ASAP primitives** — decide whether and how each intent can be answered by a summary,
and select/size the corresponding summary family.
The [documentation map](docs/README.md) gives each audience a complete reading path.

## Building
## Build

```sh
cargo build
cargo test --workspace
```
cargo build
cargo test --workspace

No external setup required. See [`docs/user-guide/user-guide.md`](docs/user-guide/user-guide.md) for how to run a
query through the pipeline.

## Glossary

Let us define a few terms.

### Query language

E.g. PromQL, Clickhouse dialect of SQL, Datafusion dialect of SQL, etc.

### Query intent

The semantics of what the query wants to do. Multiple different query strings (in the same language or different) can have the same query intent.

See examples below. All queries in the same example share the same query intent.
In each example, the `orders` table has columns `time`, `price`, `city`, and `category`.

#### Example 1

```SQL
SELECT SUM(price)
FROM orders
WHERE time BETWEEN NOW() and NOW() - 1m
GROUP BY city
```

and

```SQL
WITH intermediate_table AS (
SELECT SUM(price)
FROM orders
WHERE time BETWEEN NOW() and NOW() - 1m
GROUP BY city, category
)
SELECT SUM(price)
FROM intermediate_table
GROUP BY city
```

#### Example 2

```SQL
SELECT SUM(price)
FROM orders
WHERE time BETWEEN NOW() and NOW() - 1m
GROUP BY city
```

and

```promql
sum by (cpu) (sum_over_time(orders[1m]))
```

### Query workload

A set of queries that are to be executed. For now, this is either a batch of queries executed on data at rest, or a set of repeating queries executed on recently ingested data.

### Pre-ASAP IR

A common IR that different query workloads in different languages are normalized to. has **no notion** of ASAP primitives or summaries. The purpose of this IR is to simply have a common representation for diverse query workloads and languages.

### Post-ASAP IR

An IR that includes of ASAP primitives, apart from the usual relational and time-series operators.

### Plan

A DAG constructed in either the pre-ASAP IR or post-ASAP IR. Pre-ASAP plan represents the original exact intent of the input query workload. Post-ASAP plan represents the same intent using ASAP primitives.

## Scope

As of Aug 13, 2026, ASAPPlanner will be scoped to:
- generating a set of candidate ASAP plans, not choosing the optimal one between them
- not caring about CTSA stages i.e. whether a part of a plan is executed at the collector or at the analytics stage
- not caring about assignment of physical resources, like CPU threads and memory, to nodes in the ASAP plan i.e. ASAPPlanner is NOT doing any phyiscal query planning (ref: database term)
- // TODO: add scope on what "IR to ASAP plan logic" we support right now

---

## High-level workflow

```text
query workload
│ parse
pre-ASAP plan
│ canonicalize
canonical pre-ASAP plan
│ ASAP-aware mapping
post-ASAP plan
```

# Example: Equivalent SQL and PromQL have the same pre-ASAP plan

## SQL

```sql
SELECT service, COUNT(*)
FROM metrics
WHERE region = 'us-east'
GROUP BY service
ORDER BY COUNT(*) DESC
LIMIT 10;
```

## PromQL

```promql
topk(
10,
count by (service) (
{region="us-east"}
)
)
```

## Unified intent

```text
Aggregate(
reduction = Reduce(by = [service]),
measures = [TopK { k = 10 }],
child = Filter(
pred = region = "us-east",
child = Scan("metrics")
)
)
```

The two languages may use very different syntax and data models, but the summary-relevant
semantic intent is the same.

# End-to-end example

Consider:

```sql
SELECT service, COUNT(*)
FROM metrics
WHERE region = 'us-east'
GROUP BY service
ORDER BY COUNT(*) DESC
LIMIT 5;
```

## parse + canonicalize

```text
Aggregate(
reduction = Reduce(by = [service]),
measures = [TopK { k = 5 }],
child = Filter(
pred = region = "us-east",
child = Scan("metrics")
)
)
```

Equivalent PromQL converges to the same structure.

## ASAP-aware mapping

```text
Aggregate(reduction = Reduce(by = [service]), measures = [TopK { k = 5 }], ...)
SpaceSaving(k=5)
```

# Design principles

## 1. Normalize semantics, not syntax

The canonical algebra should represent query intent rather than reproduce the source
language's syntax tree. Canonicalization should make semantically equivalent computations structurally identical so
that reusable sub-computations can be recognized and shared.

## 2. Make summary-relevant intents explicit

Operations such as `Quantile`, `DistinctCount`, and `TopK` deserve semantic representation
because they have distinct summary mappings.

## 3. Avoid adding nodes prematurely

Do not add a node merely because SQL has an operator with that name. Add a node when it carries
semantic information that matters downstream.

# Immediate TODOs

- Change input from a single query string to `QueryWorkload` (tracked in [#194](https://github.com/ProjectASAP/ASAPPlanner/issues/194)
- Remove legacy data structures and types (tracked in [#179](https://github.com/ProjectASAP/ASAPPlanner/issues/179), [#205](https://github.com/ProjectASAP/ASAPPlanner/issues/205)
- Implement the ASAP-aware mapping [logic and interfaces](docs/design_docs/asap-aware-mapping/README.md)
- Connect output of ASAPPlanner to asap-fusion
- Connect output of ASAPPlanner to ASAPCollector and ASAPQuery (see open question #1 below)

# Next steps

- [Description of pre-ASAP IR](docs/design_docs/pre-asap-ir.md)
- [Description of post-ASAP IR](docs/design_docs/post-asap-ir.md)
- [Converting a QueryWorkload to a pre-ASAP plan](docs/design_docs/parse_and_canonicalize.md)
- [Converting a pre-ASAP plan to a post-ASAP plan](docs/design_docs/asap-aware-mapping/README.md)
- [ASAPPlanner and downstream application boundaries](docs/design_docs/asapplanner-downstream-boundary.md)
- [Guide on how to use ASAPPlanner](docs/user-guide/user-guide.md)
- [Public library functions and controls](docs/developer_docs/library-api.md)

# Open questions

1. **Integration with downstream artifacts:** Implement the documented
[ASAPPlanner and downstream application boundary](docs/design_docs/asapplanner-downstream-boundary.md)
so one backend compilation produces consistent Collector, backend, and
query-plan projections from the selected logical Post-ASAP DAG.
2. **Grouping semantics:** Should grouping remain embedded in `Aggregate`, or should grouping
become a reusable relational dimension node?
3. **Expression semantics:** Which arithmetic or derived expressions need dedicated semantic
nodes because they materially affect summary selection?
4. **Approximation contracts:** Should accuracy/error requirements be fields on the intent,
the workload, or the measure itself?
5. **Summary composability:** How should nested intents describe summaries that can be merged,
transformed, or reused across queries?
No external setup is required for the standard build and test suite.
2 changes: 1 addition & 1 deletion crates/asap-aware-mapping/src/empirical_cost.rs
Original file line number Diff line number Diff line change
Expand Up @@ -478,7 +478,7 @@ mod tests {
.implementation
.contains("SYNTHETIC"));
let schema: serde_json::Value = serde_json::from_str(include_str!(
"../../../docs/developer_docs/offline-sketch-evidence.schema.json"
"../../../docs/develop_docs/offline-sketch-evidence.schema.json"
))
.unwrap();
assert_eq!(
Expand Down
2 changes: 1 addition & 1 deletion crates/types/src/pre_asap/cse.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
//!
//! CSE only runs on an already-bound, already-canonicalized tree —
//! structural matching is meaningless before canonicalization has converged
//! semantically-equivalent queries onto one shape (`docs/design_docs/pre-asap-ir.md`
//! semantically-equivalent queries onto one shape (`docs/develop_docs/pre-asap-ir.md`
//! design principle 3; `median(latency)` and `approx_percentile_cont(latency,
//! 0.5)` already lower to an identical `AggIntent::Quantile` today, per
//! `sql_lowering.rs`'s `median_is_the_same_intent_as_an_explicit_half_percentile`
Expand Down
32 changes: 32 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# ASAPPlanner documentation

Choose the path that matches what you need to do.

## New to ASAPPlanner

1. Read the [planner pipeline](design_docs/concepts/planner-pipeline.md).
2. Use the [glossary](design_docs/concepts/glossary.md) when a term is unfamiliar.
3. Learn the roles of the [Pre-ASAP IR](design_docs/concepts/pre-asap-ir.md) and [Post-ASAP IR](design_docs/concepts/post-asap-ir.md).
4. Run one query with [the inspection guide](user_guide_docs/run-a-query.md).

## Run or inspect a query

Follow [Run and inspect a query](user_guide_docs/run-a-query.md). Tool-specific setup stays with the tool, including the [DAG viewer instructions](../tools/dag-viewer/RUNNING.md).

## Embed the library

Use [Public library functions and examples](develop_docs/library-api.md) for
frontend lowering, workload search, ranking, optional selection and lifecycle
integration.

## Extend the planner

1. Read the [ASAP-aware mapping architecture](develop_docs/asap-aware-mapping-architecture.md).
2. Consult [mapping contracts](develop_docs/asap-aware-mapping-contracts.md).
3. Follow [Extend ASAP-aware mapping](develop_docs/extend-asap-aware-mapping.md).

## Understand a design

- [Design documentation](design_docs/README.md) describes concepts, architecture, decisions, and proposals.
- [Developer documentation](develop_docs/README.md) specifies current interfaces, IR nodes, formats, evidence, and extension workflows.
- [User guides](user_guide_docs/README.md) show supported commands and verification steps.
Loading
Loading