From 75ccba7c1667986ca20f4696bb57c9e1e52170a9 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 04:27:40 +0000 Subject: [PATCH] docs: reorganize backend design guides --- .../asapquery-compatibility-profile.md | 578 ++++++------------ .../continuous-summary-completeness.md | 125 +++- .../empirical-o11y-execution-plan.md | 150 +++-- docs/design_docs/shape-aware-erp-v1.md | 186 ++++-- 4 files changed, 521 insertions(+), 518 deletions(-) diff --git a/docs/design_docs/asapquery-compatibility-profile.md b/docs/design_docs/asapquery-compatibility-profile.md index e911510fd..d1ba13372 100644 --- a/docs/design_docs/asapquery-compatibility-profile.md +++ b/docs/design_docs/asapquery-compatibility-profile.md @@ -1,438 +1,228 @@ # ASAPQuery compatibility profile -> Status: implemented; wider maintenance capabilities retain separate admission gates +> Status: implemented; wider maintenance capabilities retain separate admission gates. > > Reference: [ProjectASAP/ASAPQuery at `9fb051a`](https://github.com/ProjectASAP/ASAPQuery/tree/9fb051aa798361fca8e3012835412cb6fa338a0c) -> -> Scope: a deliberately smaller target operating profile of ASAPQuery-backend that -> accepts raw Prometheus Remote Write samples and does not depend on -> ASAPCollector. -## Goal +The `asapquery` profile is a strict, startup-validated subset of +ASAPQuery-backend. It accepts raw Prometheus Remote Write samples, builds +summaries in process, serves supported PromQL and falls back to Prometheus. It +does not depend on ASAPCollector. + +## Document map + +1. [Profile at a glance](#profile-at-a-glance) +2. [Worked example](#worked-example) +3. [Profile boundary](#profile-boundary) +4. [Planning and runtime contracts](#planning-and-runtime-contracts) +5. [Remote Write contract](#remote-write-contract) +6. [Query and fallback contract](#query-and-fallback-contract) +7. [Activation and readiness](#activation-and-readiness) +8. [Compatibility evidence](#compatibility-evidence) +9. [Completion and extensions](#completion-and-extensions) + +## Profile at a glance + +```mermaid +flowchart LR + P[Prometheus] -->|remote_write| I[Backend ingest] + I --> C[In-process precompute] + C --> S[SummaryStore] + U[PromQL client] --> Q[Query router] + Q -->|ready and supported| S + Q -->|fallback| P +``` -ASAPQuery-backend has a broader architecture than ASAPQuery: it can coordinate -external collectors, accept materialized summaries over modified OTLP, use -multiple storage tiers, and compile distributed physical plans. This profile -does not remove those capabilities. It defines the target configuration required -for ASAPQuery-compatible behavior and gives that configuration an independent -end-to-end acceptance target. Selecting `--profile asapquery` is now a strict, -startup-validated subset of the broader runtime. +Prometheus remains the exact raw-data authority. The backend accelerates only +installed queries whose summary state is complete, fresh and accurate enough. +Unsupported, unsafe, stale or not-yet-ready queries follow the explicit fallback. -The user-visible goal is the same drop-in shape as ASAPQuery: +Planning uses configured `QueryWorkload` and `DataWorkload`, canonical +ASAPPlanner selection and a backend-only physical compiler: ```text -Prometheus ── remote_write raw samples ──► ASAPQuery-backend - ▲ │ - │ exact fallback │ in-process summaries - │ ▼ -Grafana / PromQL client ── query ──► accelerated query endpoint +workloads -> ASAPPlanner -> selected candidate -> backend compiler + -> SummaryCatalog + -> PrecomputePlan + -> QueryPlan ``` -Prometheus remains the exact raw-data system. It sends a copy of ingested -samples to ASAPQuery-backend through Prometheus Remote Write. The backend builds -planned summaries from those raw samples and serves supported PromQL from the -summaries. Unsupported, not-yet-ready, stale, or unsafe queries are forwarded -to Prometheus. - -The reference behavior is anchored by ASAPQuery's -[top-level architecture](https://github.com/ProjectASAP/ASAPQuery/blob/9fb051aa798361fca8e3012835412cb6fa338a0c/README.md), -[Remote Write decoder](https://github.com/ProjectASAP/ASAPQuery/blob/9fb051aa798361fca8e3012835412cb6fa338a0c/asap-query-engine/src/drivers/ingest/prometheus_remote_write.rs), -[query tracker](https://github.com/ProjectASAP/ASAPQuery/blob/9fb051aa798361fca8e3012835412cb6fa338a0c/asap-query-engine/src/query_tracker/tracker.rs), -and -[precompute design](https://github.com/ProjectASAP/ASAPQuery/blob/9fb051aa798361fca8e3012835412cb6fa338a0c/asap-query-engine/src/precompute_engine/precompute_engine_design_doc.md). -The goal is behavioral compatibility, not copying its historical planner or -internal types. +The three outputs share plan and materialization identities and activate as one +immutable version. -## Profile boundary +## Worked example -### Required in this profile - -- Prometheus Remote Write v1 ingestion at `POST /api/v1/write`; -- Snappy decompression and protobuf `WriteRequest` decoding; -- raw scalar sample, stale-marker, and label canonicalization; -- in-process streaming precompute with windowing and lateness handling; -- an in-process summary store sufficient for the accelerated query path; -- Prometheus-compatible instant and range query endpoints; -- startup-supplied `QueryWorkload` and `DataWorkload` snapshots and canonical - ASAPPlanner invocation; -- backend-only physical compilation and atomic plan activation; -- summary-backed execution with explicit Prometheus fallback; and -- one self-contained compatibility demo and end-to-end test. - -### Excluded from the MVP profile - -- ASAPCollector discovery, configuration, or plan publication; -- `CollectorPlan`, `SDKPlan`, OpAMP, or collector activation evidence; -- OTLP or modified-OTLP ingestion; -- prebuilt summary ingestion from external producers; -- source sampling, GOS, sparse delta transmission, and frame ACK protocols; -- VictoriaMetrics Remote Write, Kafka, CSV, JSON, or other ingest connectors in - the MVP; -- SQL, ClickHouse, and Elasticsearch query protocols in the MVP; -- S3, Thanos, Gorilla, or other durable/cold tiers; and -- distributed placement or multi-producer summary merging. - -Excluded features may continue to exist in the larger product. They must be -disabled and must not be startup dependencies when this profile is selected. - -## Target architecture +An operator configures a repeating five-minute sum query and Prometheus sends raw +samples to the backend: -```text -Runtime data path - -Prometheus ── remote_write ──► raw receiver ──► precompute ──► SummaryStore - -PromQL client ──► Prometheus API ──► query router - ├──► summary readout ◄── SummaryStore - └──► exact fallback ──► Prometheus - -Planning path - -configured QueryWorkload + DataWorkload - │ - ▼ - ASAPPlanner - Post-ASAP candidates + selection - │ - ▼ - backend-only physical compiler - │ │ │ - ▼ ▼ ▼ - PrecomputePlan SummaryCatalog QueryPlan DAG - │ │ │ - ▼ ▼ ▼ - precompute catalog SID-bound executor +```yaml +profile: asapquery +prometheus_fallback: http://prometheus:9090 +query_workload: + - id: api-request-sum + expression: sum_over_time(api_requests_total[5m]) + every: 1m +data_workload: + metric: api_requests_total + ingestion_rate: 10000 ``` -The plan has no collector projection. One compile produces an atomic -`PhysicalPlan` containing one `SummaryCatalog` plus sibling backend-local -`PrecomputePlan` and `QueryPlan` sections from the same selected Post-ASAP candidate. -`PrecomputePlan` directly is the runtime precompute contract; there is no second -streaming-config semantic model or lossy conversion step. All sections share -plan and materialization identities. -One immutable version installs the precompute configuration, store catalog, and -inactive query routes atomically. Materialization readiness is runtime state: -each route becomes eligible for summary serving only after its required windows -have complete and fresh coverage. +Startup compiles a backend-local materialization, installs inactive query routes +and begins accepting Remote Write. Before five minutes of complete coverage, the +query is forwarded to Prometheus. When the materialization becomes ready, the +same request is served from SummaryStore. An unsupported expression such as +`absent(up)` continues to fall back. -## Component responsibilities +```text +request -> active QueryPlan? + -> supported binding? + -> complete and fresh state? + yes: summary result + no: semantically equivalent Prometheus request +``` -### ASAPPlanner +## Profile boundary -ASAPPlanner consumes the complete query workload and associated data workload. -It owns query semantics, workload-wide sharing, abstract summary candidates, -accuracy reasoning, logical window/lifecycle choices, selection, and exact -fallback decisions. +Required: -The compatibility profile consumes canonical types and behavior from the pinned -ASAPPlanner revision. It must not restore ASAPQuery's historical planner as a -second planner or copy Planner optimizer, intent-algebra, or sketch-capability -rules into ASAPQuery-backend. +- `POST /api/v1/write` Remote Write v1 with Snappy/protobuf decoding; +- raw scalar samples, stale markers and canonical labels; +- backend-local streaming precompute, windowing and lateness handling; +- in-process SummaryStore and Prometheus-compatible instant/range endpoints; +- startup workload snapshots, canonical Planner invocation and physical compile; +- atomic activation, readiness gating and explicit Prometheus fallback; +- a self-contained compatibility demo and end-to-end test. -### ASAPQuery-backend control plane +Disabled and not startup dependencies: -For the first MVP, operators provide immutable `QueryWorkload` and -`DataWorkload` snapshots at startup. The control plane: +- ASAPCollector, CollectorPlan, SDKPlan, OpAMP and collector activation; +- OTLP/prebuilt-summary ingestion and external summary producers; +- sampling, GOS, sparse delta transmission and frame ACK protocols; +- non-Prometheus ingest adapters and non-PromQL query protocols; +- durable/cold tiers and distributed placement or merging. -1. validates and loads both configured workload snapshots; -2. invokes ASAPPlanner with the complete `QueryWorkload` and associated - `DataWorkload`; -3. enumerates only backend-local implementations for Planner candidates; -4. returns implementation-cost evidence needed for selection; -5. compiles the selected candidate into a matching SummaryCatalog, PrecomputePlan, - and QueryPlan; and -6. stages and atomically activates those views. +These capabilities may exist in broader ASAPQuery-backend profiles. -It does not enumerate SDK or Collector placements in this profile. A Planner -candidate that has no backend-local SummaryStore materialization and readout -implementation is unavailable; the control plane must not assign it an -optimistic zero cost. +## Planning and runtime contracts -Online query observation, data-workload estimation from Remote Write, and -replanning may be added later. If enabled, the data-plane query endpoint emits -bounded canonical observations and the control plane aggregates them; the -request path never owns planning. These capabilities are not startup or MVP -dependencies. +| Component | Responsibility | +| --- | --- | +| ASAPPlanner | Query semantics, sharing, abstract candidates, accuracy, lifecycle choices and exact fallback | +| Control plane | Load workloads, advertise backend-local implementations/costs, compile and activate one physical version | +| Remote Write adapter | Decode and validate wire input; emit canonical raw samples | +| Precompute runtime | Route series, maintain windows/accumulators and publish state | +| SummaryStore | Index state by materialization, labels and logical window; track coverage/readiness | +| Query path | Execute the installed QueryPlan or forward an equivalent request to Prometheus | -### Raw ingest and precompute +The backend does not restore ASAPQuery's historical planner or precompute engine. +Historical fixes remain a migration source and require regression tests or an +explicit inapplicability record. The audit includes idle/trailing-window closure, +wall-clock safety, pane eviction, millisecond windows, value routing, +CMS-with-heap parameters and accumulator routing. -The Remote Write adapter owns only wire decoding and validation: +A candidate without backend-local materialization and readout support is +unavailable; it never receives optimistic zero cost. Request handling does not +perform planning or choose a substitute materialization. -```text -HTTP body - -> verify request limits and content encoding - -> Snappy decode - -> protobuf WriteRequest decode - -> validate __name__, labels, timestamps, and sample encodings - -> recognize the Prometheus stale-NaN marker before ordinary numeric checks - -> canonical series key - -> route raw samples to the active PrecomputePlan -``` +## Remote Write contract -The precompute engine owns series routing, bounded buffering, watermark and -lateness behavior, window assignment, selected accumulator updates, and writes -to SummaryStore. The decoder must not choose an aggregation family. - -The current ASAPQuery-backend streaming precompute engine is the destination, -but the ASAPQuery history is a required bug-fix migration source. Before the -compatibility profile is complete, relevant ASAPQuery precompute fixes must be -audited commit by commit and either migrated with regression tests or recorded -as inapplicable because the affected feature was intentionally retired. The -known audit set includes idle/trailing-window closure, active-ingest wall-clock -safety, move-out at pane eviction, millisecond windows, value-column routing, -CMS-with-heap parameter parsing, and accumulator-family routing. - -Restoring Remote Write must add an adapter to the current engine; it must not -restore or fork the historical ASAPQuery engine. In particular, retired -`SetAggregator` and `DeltaSetAggregator` paths are not restored merely because -their historical patches touched precompute code. Compatibility tests preserve -both migrated ASAPQuery fixes and newer backend fixes, and expose any remaining -semantic mismatch rather than weakening the expected results. - -The exact Prometheus stale-NaN bit pattern is recognized, deduplicated and -counted, then excluded from numeric aggregation. It is not currently propagated -as a worker lifecycle event. Other non-finite values, native histograms and -exemplars are rejected. Only Remote Write v1 is supported. - -Within the configured in-memory deduplication horizon, an identical canonical -series/timestamp/value is a duplicate; a conflicting value is rejected. The -receiver validates the batch and reserves all bounded worker-queue capacity -before exposing messages. A `204` acknowledges that queue admission and -in-memory dedup bookkeeping, not completed accumulator mutation or durable -commit. There is no durable raw-sample WAL or receiver dedup recovery after a -process restart. Summary persistence is a separate later stage. - -Invalid batches fail before enqueueing. Queue/dedup capacity exhaustion returns -a retryable `503`; body/decoded size limits return `413`. The dedup horizon must -cover configured lateness and expected retry duration, but it does not provide -crash-safe exactly-once delivery. Prometheus remains the raw-data authority. - -Remote Write carries neither a producer-partition roster nor an authoritative -watermark. Finite `POST /api/v1/precompute/drain` closes the input generation; -it is not continuous completion. Existing typed `SummaryWatermarkBarrier` and -coordinator APIs need registered producer/partition identity and publication -ordering before they can establish continuous closure. There is no public HTTP -barrier endpoint in this profile. - -### SummaryStore - -The MVP may use the existing in-process summary store. It stores only state -produced by the active backend-local PrecomputePlan and indexes it by -materialization, canonical label values, and logical window. It tracks enough -coverage and watermark state to distinguish ready, missing, incomplete, and -stale ranges. - -Persistent/cold storage is an extension of ASAPQuery-backend, not a dependency -of this compatibility profile. A process restart may lose the warm summary -cache only if query routing falls back to Prometheus until the new plan has -rebuilt complete coverage. Recovery creates a new backend-local producer/runtime -epoch. Pre-crash partial windows remain incomplete and must never be combined -with post-restart samples; they can be served only after explicit Prometheus -backfill, otherwise the first eligible result is a complete post-restart window. - -### Query path - -The public MVP query surface is: +The adapter performs: ```text -GET or POST /api/v1/query -GET or POST /api/v1/query_range +limits -> Snappy decode -> WriteRequest decode -> sample/label validation + -> stale-marker recognition -> canonical series key -> active plan routing ``` -For each request, the Prometheus adapter preserves query semantics and response -shape. Semantic preservation includes `query`, `time`, `start`, `end`, `step`, -and `timeout`, plus configured tenant and authorization context; it does not -require byte-for-byte reproduction of the incoming HTTP request. Routing has two -successful outcomes: +Only v1 scalar samples are supported. The exact Prometheus stale-NaN marker is +recognized, deduplicated and counted, then excluded from numeric aggregation. +Other non-finite values, native histograms and exemplars are rejected. -1. execute the active QueryPlan DAG, whose materialization bindings were - resolved through SummaryCatalog, when compatible summary state has - complete and fresh coverage; or -2. forward a semantically equivalent request to the configured Prometheus - endpoint. +Within the in-memory dedup horizon, identical series/timestamp/value input is a +duplicate and a conflicting value is rejected. The receiver validates the batch +and reserves all queue capacity before enqueueing. A `204` acknowledges admission +and dedup bookkeeping, not accumulator mutation or durable commit. -A parse failure, store miss, unsupported expression, inactive plan, incomplete -window, stale coverage, or insufficient accuracy is not an empty successful -summary result. It follows the explicit fallback route or returns an error if -fallback is unavailable. +| Condition | Result | +| --- | --- | +| Invalid batch | Fail before enqueueing | +| Queue/dedup capacity exhausted | Retryable `503` | +| Body or decoded size limit exceeded | `413` | +| Process restart | No raw WAL or dedup recovery; Prometheus remains authority | -The first compatibility level is intentionally explicit rather than claiming -all PromQL. It supports raw scalar samples, canonical label grouping, tumbling -window sum, Prometheus `rate` and `increase` over counters, one sketch-backed -quantile operation, and instant and range evaluation of those planned -summaries. Selectors or expressions outside that set exercise the exact -Prometheus fallback path. Expanding the accelerated surface requires a versioned -compatibility-level change and conformance tests. +Remote Write provides no producer roster or authoritative watermark. Finite +`POST /api/v1/precompute/drain` closes one input generation; it does not prove +continuous completion. See [summary completeness](continuous-summary-completeness.md). -## Planning and activation lifecycle +## Query and fallback contract -Startup is fallback-safe: +The public surface is: ```text -start backend - -> verify Prometheus fallback health - -> load configured QueryWorkload and DataWorkload snapshots - -> run Planner candidate search and selection - -> compile one PhysicalPlan (SummaryCatalog + PrecomputePlan + QueryPlan) - -> atomically install precompute + catalog + inactive routes under one version - -> accept Remote Write and forward every query to Prometheus - -> enter Materializing state - -> wait for complete summary coverage - -> mark each ready materialization Serving +GET or POST /api/v1/query +GET or POST /api/v1/query_range ``` -The lifecycle is `Compiled -> Installed -> Materializing -> Ready -> Serving`. -Installation is the atomic configuration boundary; readiness is evidence about -runtime data coverage. Plan replacement builds a new immutable snapshot. -Precompute, store metadata, and query routing must never observe a mixture of -old and new aggregation parameters. Until the new plan is installed and warm, -the previous compatible route or Prometheus remains authoritative. - -The first MVP plans once from the startup snapshots. Runtime observation and -repeated replanning are optional, but any later implementation must preserve the -same atomic cutover and warmup rules. - -## Relationship to the broader backend - -| Concern | ASAPQuery compatibility profile | Broader ASAPQuery-backend | -| --- | --- | --- | -| Ingest source | Prometheus Remote Write raw samples | Collector materializations over modified OTLP and other explicit profiles | -| Summary construction | Backend-local only | Collector or backend placement | -| Physical outputs | SummaryCatalog + PrecomputePlan + QueryPlan | SummaryCatalog/CollectorPlan/PrecomputePlan/TransmissionPlan/QueryPlan views as applicable | -| Query protocol | Prometheus HTTP / PromQL | Additional protocols may be supported | -| Exact fallback | Upstream Prometheus | Prometheus or another compiled storage/query route | -| Storage required for MVP | In-process warm summary state | Warm, durable, archive, and remote tiers | -| Sampling and delta | Disabled | Optional physical mechanisms | - -The compatibility profile is a restricted runtime configuration. Every enabled -component belongs to ASAPQuery-backend; broader distributed features remain -available only outside this profile. - -The profile is also not a literal subset of historical ASAPQuery internals. It -preserves the relevant external behavior while adding the current canonical -ASAPPlanner types, versioned physical compilation, readiness evidence, and -stronger activation and retry contracts. - -## Implementation status - -| Area | Implemented contract | Executable evidence | -| --- | --- | --- | -| Startup/profile | Collector-free startup, excluded-component validation, fallback health gate | `data_plane` profile tests and production-process E2E | -| Physical planning | Canonical workload snapshot to one atomic SummaryCatalog/PrecomputePlan/QueryPlan bundle | `compatibility_demo_snapshot_compiles_the_complete_query_matrix` | -| Remote Write | Strict v1 decoding, stale-marker exclusion, limits, bounded in-memory deduplication and backpressure | receiver unit tests plus process E2E replay/corrupt-batch assertions | -| Precompute/store | Raw samples use the planned family; first catch-up batches close all complete windows; sketch and exact payloads share canonical SID semantics | worker/store tests and four-family process matrix | -| Query execution | QueryPlan-only serving-time lookup, node-level materialization binding, generic DAG traversal, exact fallback | instant/range process matrix and fallback request capture | -| Atomic activation | Versioned stage/activate snapshot and materialization readiness state | physical-plan endpoint tests and `/physical-plan/status` assertions | -| Real deployment | Prometheus remote_write with no Collector | `./scripts/e2e.sh asapquery-demo` | - -## Implemented phases +The adapter preserves `query`, `time`, `start`, `end`, `step`, `timeout`, tenant +and authorization semantics. It either executes the active QueryPlan with exact +catalog bindings or forwards a semantically equivalent request to Prometheus. -### Phase A: profile and startup contract +Parse failure, unsupported expressions, inactive plans, store misses, incomplete +or stale windows and insufficient accuracy are never successful empty summary +results. They fall back or return an error when fallback is unavailable. -Add an explicit `asapquery` profile with startup validation. It permits only -Prometheus fallback, Remote Write ingestion, PromQL HTTP serving, backend-local -precompute, the warm summary store, and configured `QueryWorkload` and -`DataWorkload` inputs. An excluded connector or required Collector endpoint is -a configuration error. +The first compatibility level covers raw scalar samples, canonical grouping, +tumbling-window sum, Prometheus `rate` and `increase`, one sketch-backed quantile, +and instant/range evaluation. Other expressions fall back. Expanding this set +requires a versioned compatibility level and conformance tests. -Acceptance: the backend starts with no ASAPCollector or OTLP endpoint and all -queries initially reach Prometheus. +## Activation and readiness -### Phase B: Remote Write ingestion - -Implement the v1 receiver, strict resource limits, canonical label and stale -marker handling, bounded deduplication/retry behavior, visible backpressure, and -routing into the existing raw-sample precompute input. - -Acceptance: valid Snappy/protobuf batches produce the same canonical samples and -stale-marker recognition as a reference decoder; replayed whole or partial batches -converge without double-counting; corrupt, oversized, conflicting, and -overloaded requests cannot leave untracked mutations while returning success. - -### Phase C: backend-only planning - -Load the configured query and data workload snapshots, call the pinned -ASAPPlanner, enumerate backend-local implementations, and compile one atomic -PhysicalPlan with SummaryCatalog, PrecomputePlan, and QueryPlan. Do not create or -wait for CollectorPlan. - -Acceptance: captured Planner input, selected Post-ASAP candidate, -SummaryCatalog, PrecomputePlan, and QueryPlan are deterministic golden artifacts -with matching plan/materialization/window/family/parameter identities. - -### Phase D: atomic activation and warmup - -Install the precompute configuration, store catalog, and inactive query routes -atomically under one plan version. Track readiness separately for each -materialization and keep its queries on Prometheus until all required windows -are complete and fresh. - -Acceptance: injected failure at every installation boundary exposes either the -old complete configuration or the new complete configuration, never a mixed -one. An installed-but-materializing plan remains on fallback and cannot be -mistaken for a serving route. - -### Phase E: PromQL serving and fallback - -Serve the declared compatibility-level instant and range queries from planned -summaries. Forward every unsupported or unsafe request to Prometheus with -semantically equivalent parameters and configured request context, and preserve -Prometheus response types, labels, timestamps, warnings, and errors. - -Acceptance: accelerated results satisfy their declared error bound against -Prometheus, exact operations match Prometheus semantics, and fallback responses -are equivalent to direct Prometheus calls. - -### Phase F: compatibility demo - -`./scripts/e2e.sh asapquery-demo` runs Prometheus with `remote_write` configured -to the backend, starts the backend with fixed workload snapshots, sends their -corresponding repeating queries -through the backend, wait for planning and warmup, and capture route decisions -and resource measurements. - -Acceptance evidence includes: +```text +verify fallback -> load workloads -> select -> compile + -> atomically install catalog + precompute + inactive query routes + -> accept writes and fall back queries + -> materialize complete coverage + -> mark each eligible route serving +``` -- Remote Write requests, samples, rejected requests, duplicates, and bytes; -- configured workload snapshots and the exact Planner input/output artifacts; -- active plan/materialization identities and activation time; -- summary versus fallback query counts; -- window coverage and end-to-end freshness; -- result error against direct Prometheus; and -- query latency, backend CPU, and backend memory before and after activation. +The lifecycle is `Compiled -> Installed -> Materializing -> Ready -> Serving`. +Installation is the atomic configuration boundary; readiness describes runtime +coverage. A replacement never exposes mixed parameters from old and new plans. +Until the new version is warm, the previous compatible route or Prometheus is +authoritative. + +After restart, pre-crash partial windows remain incomplete and cannot combine +with post-restart samples. They require explicit backfill; otherwise serving +starts with the first complete post-restart window. + +## Compatibility evidence + +| Area | Required executable evidence | +| --- | --- | +| Startup | Collector-free profile validation and fallback health gate | +| Planning | Deterministic workload, selected candidate, catalog and two backend plans | +| Remote Write | Reference decoding, stale markers, limits, dedup and backpressure | +| Precompute/store | Planned families, complete windows and canonical state identity | +| Query | Instant/range summary execution plus captured fallback requests | +| Activation | Stage/activate failure tests and readiness assertions | +| Deployment | `./scripts/e2e.sh asapquery-demo` with no Collector | + +The E2E query matrix compares every backend result with the same Prometheus: + +| Case | Instant | Range | +| --- | --- | --- | +| `rate(counter[window])` | Required | Every returned step | +| `increase(counter[window])` | Required | Every returned step | +| Planned sum | Required | Required | +| Planned sketch quantile | Required | Required | +| Unsupported expression | Exact fallback | Exact fallback | -The E2E query matrix includes all of the following through the backend and -compares each result with a direct request to the same Prometheus instance: +Counter fixtures cover reset, irregular spacing and boundary samples. Assertions +include values, labels, timestamps, result type and range-step count; HTTP success +alone is insufficient. The run records workload/Planner artifacts, active IDs, +route decisions, coverage, freshness, error, latency, CPU and memory. -| PromQL case | Instant `/api/v1/query` | Range `/api/v1/query_range` | -| --- | --- | --- | -| `rate(counter[window])` | Required | Required, including every returned step | -| `increase(counter[window])` | Required | Required, including every returned step | -| planned sum | Required | Required | -| planned sketch-backed quantile | Required | Required | -| unsupported expression | Exact fallback required | Exact fallback required | - -The counter fixture includes monotonic input, at least one counter reset, -irregular sample spacing, and samples near window boundaries. The assertions -cover values, labels, timestamps, result type, and range-step count. A test that -only proves that the endpoint returns HTTP success does not satisfy this matrix. - -## Post-MVP compatibility extensions - -The first query extension should be a SQL endpoint because ASAPQuery exposed SQL -as an additional query surface. It must translate SQL into canonical Planner -query semantics and reuse the same catalog-backed QueryPlan readiness, summary store, and -exact-fallback rules; it must not introduce a second planner or separately -configured materializations. Its supported SQL subset and fallback target need -their own versioned compatibility contract and conformance cases. - -CSV and JSON ingestion can follow as convenience adapters. Each adapter is -limited to parsing and validation before emitting the same canonical raw-sample -records as Remote Write. It must not choose aggregation families or bypass the -active PrecomputePlan. These adapters remain optional and cannot become startup -dependencies of the Remote Write profile. - -## MVP completion criterion +## Completion and extensions The executable completion command is: @@ -440,13 +230,11 @@ The executable completion command is: ./scripts/e2e.sh asapquery-demo ``` -It starts Prometheus and ASAPQuery-backend without ASAPCollector, ingests only -through Prometheus Remote Write, plans from the configured workloads, activates a -backend-local summary, serves both the declared sum and sketch-backed quantile -compatibility cases plus Prometheus `rate` and `increase` through both instant -and range endpoints from complete summary windows, and transparently falls back -for an unsupported query. The run must fail if ingestion, planning, activation, -coverage, accuracy, counter-reset handling, range-step equivalence, or fallback -evidence is missing. +It must fail when ingestion, planning, activation, coverage, accuracy, counter +semantics, range equivalence or fallback evidence is missing. Starting components +or exposing `/api/v1/write` alone is not completion. -Starting the components or exposing `/api/v1/write` alone is not completion. +SQL is the first intended query extension. It must translate into canonical +Planner semantics and reuse the same catalog, readiness, store and fallback +contracts. Optional CSV/JSON adapters may emit the same canonical raw-sample +records but cannot choose aggregations or bypass PrecomputePlan. diff --git a/docs/design_docs/continuous-summary-completeness.md b/docs/design_docs/continuous-summary-completeness.md index 34549b8df..7558f11ca 100644 --- a/docs/design_docs/continuous-summary-completeness.md +++ b/docs/design_docs/continuous-summary-completeness.md @@ -1,15 +1,126 @@ # Summary publication completeness for accepted input -The SummaryStore owns an in-memory admission inventory keyed by catalog generation, summary definition, group population and physical time window. Remote Write reserves every worker queue slot before admitting any coordinates, then sends the same immutable input revision with the queued work. Queue rejection leaves neither messages nor admission records behind. +Status: implemented in-memory admission and read-fence contract. Audience: +developers changing Remote Write admission, maintenance publication, SummaryStore +reads or recovery. -Workers carry the first and last consumed input revisions through materialized source and maintenance DAG outputs. State writes validate the generation and consumed revisions before modifying SummaryStore; a later output cannot acknowledge an earlier missing update. Corrections within one admitted coordinate are combined before publication. Dropped admitted input leaves the coordinate incomplete. Accepted raw-sample throughput counts once per request, without materialization fanout; materialized-output throughput counts new successful store publications, without receipt retries. +## Document map -A query rejects overlapping admitted but unpublished input. The whole QueryPlan DAG is fenced by the store revision, so branches cannot combine different publication snapshots. Both direct sketch writes (including OTLP) and exact-state writes (including SQL backfill) participate in the mutation fence; in-flight writes cannot certify a read snapshot. Finite absence proof also requires that every state mutation was admitted. The initial fence is global: unrelated updates can cause conservative exact fallback. Read-set-scoped revisions are required before claiming sustained continuous-query performance. +1. [Contract at a glance](#contract-at-a-glance) +2. [Worked example](#worked-example) +3. [Admission and publication](#admission-and-publication) +4. [Query completeness](#query-completeness) +5. [Finite drain](#finite-drain) +6. [Budgets, recovery and retirement](#budgets-recovery-and-retirement) +7. [Limits](#limits) -Finite drain closes input, waits for all workers and certifies that every accepted coordinate published. Only this closed-input proof can establish that a known series has no samples in a retained window. Missing state, unknown series, unknown time coverage and incomplete coordinates never imply an empty result. Live Remote Write provides no source watermark, so the backend does not infer global event-time completeness from the fastest series. +## Contract at a glance -Admission metadata has bounded coordinate, pending-revision and byte budgets. Completed receipts expire with configured materialization retention; pending work and the published prefixes of pending admissions remain protected. Already admitted slow-worker outputs can finish behind another worker's maintenance replay frontier. Unsolicited expired input and expired untagged replay remain rejected. +```text +Remote Write request + -> reserve all queue capacity + -> record one immutable admitted revision + -> workers publish outputs carrying consumed revisions + -> SummaryStore advances completeness only for published admitted work + -> QueryPlan reads one fenced store revision or falls back +``` -This inventory is not durable and does not establish exactly-once execution across crashes. Startup installs the authoritative catalog before persistence recovery, and version-3 persisted series metadata preserves summary definition identity and catalog provenance. Recovery requires the same catalog generation and leaves incompatible or legacy records unbound under an authoritative catalog. Reuse across changed catalog generations requires a separate explicit compatibility decision. General multi-input maintenance transforms and durable producer watermarks remain separate work. +An accepted request promises queue admission, not completed summary publication. +Missing or unpublished admitted work makes an overlapping query incomplete; it +never means an empty result. -Durable SID bindings also preserve retirement and expiry timestamps and a removal tombstone. Lifecycle changes publish through the same serialized metadata writer as the flusher before changing in-memory visibility. A stale flush snapshot cannot clear those fields. Recovery leaves removed and expired instances unregistered and preserves a still-retired instance's expiry deadline. Tombstones remain until durable state is explicitly reclaimed; this change does not claim automatic tombstone garbage collection or cross-generation reactivation. +## Worked example + +Assume request revision `101` contains samples for two coordinates: + +```text +(service=api, window=12:00..12:01) +(service=web, window=12:00..12:01) +``` + +The receiver reserves both worker slots before acknowledging the request. The +`api` worker publishes state carrying `first_revision=101` and +`last_revision=101`, while the `web` worker has not finished. + +```yaml +admission_revision: 101 +published: + api/12:00..12:01: 101 +pending: + web/12:00..12:01: 101 +``` + +A query covering both services at store fence `101` cannot combine the published +`api` state with a missing `web` state. It uses its configured exact fallback or +returns unavailability. After `web` publishes, the same fenced read may use both +states. If queue reservation initially fails, neither coordinate nor admission +record is created and the request is retryable. + +## Admission and publication + +The in-memory admission inventory is keyed by catalog generation, summary +definition, group population and physical time window. + +- Remote Write reserves every required worker slot before admitting any work. +- All queued work carries the same immutable input revision. +- Queue rejection leaves no messages or admission records. +- Workers propagate first and last consumed revisions through maintenance DAG + outputs. +- State writes validate generation and revisions before mutating SummaryStore. +- Corrections within one admitted coordinate combine before publication. +- A later output cannot acknowledge an earlier missing update. + +Accepted raw-sample throughput counts once per request, without materialization +fanout. Materialized-output throughput counts new successful publications, +without receipt retries. + +## Query completeness + +A query rejects any range overlapping admitted but unpublished input. The whole +QueryPlan DAG uses one store revision so branches cannot combine different +publication snapshots. + +Direct sketch writes, including OTLP, and exact-state writes, including SQL +backfill, participate in the mutation fence. In-flight writes cannot certify a +read snapshot. Finite absence proof additionally requires that every mutation +was admitted. + +The initial fence is global, so unrelated updates may conservatively cause exact +fallback. Read-set-scoped revisions are required before claiming sustained +continuous-query performance. + +## Finite drain + +A finite drain closes input, waits for workers and verifies that every accepted +coordinate published. Only this closed-input proof can establish that a known +series has no samples in a retained window. + +Missing state, unknown series, unknown time coverage and incomplete coordinates +never imply an empty result. Live Remote Write has no authoritative source +watermark, so the fastest series cannot establish global event-time completeness. + +## Budgets, recovery and retirement + +Admission metadata has bounded coordinate, pending-revision and byte budgets. +Completed receipts expire with materialization retention; pending work and its +published prefixes remain protected. Admitted slow-worker outputs may finish +behind another worker's replay frontier. Unsolicited expired input and expired +untagged replay are rejected. + +Startup installs the authoritative catalog before persistence recovery. Version-3 +series metadata preserves summary-definition identity and catalog provenance. +Recovery binds only the same catalog generation; incompatible and legacy records +remain unbound unless an explicit cross-generation compatibility decision exists. + +Durable SID bindings preserve retirement/expiry timestamps and removal +tombstones. Lifecycle changes use the same serialized metadata writer as the +flusher before changing in-memory visibility. Stale flush snapshots cannot clear +those fields. Recovery does not register removed or expired instances and +preserves a retired instance's expiry deadline. + +## Limits + +The admission inventory is not durable and does not provide exactly-once +execution across crashes. Tombstones remain until explicit durable-state +reclamation; automatic garbage collection is not claimed. General multi-input +maintenance transforms and durable producer watermarks remain separate work. diff --git a/docs/design_docs/empirical-o11y-execution-plan.md b/docs/design_docs/empirical-o11y-execution-plan.md index 00e98b746..81d4607ca 100644 --- a/docs/design_docs/empirical-o11y-execution-plan.md +++ b/docs/design_docs/empirical-o11y-execution-plan.md @@ -1,67 +1,109 @@ # Offline evidence and o11y planning evaluation -This is the historical offline-evidence milestone plan. The current prototype -adds actual backend-only data-plane execution using the supplied OpenMetrics -dataset, as described in the [current evaluation guide](../user_guide/o11y-replay.md). -Benchmark production and execution tooling now belong to ASAPQuery-backend; -planner-only coverage and synthetic cached-result timing are not its acceptance criteria. +Status: historical offline-evidence milestone with a current backend execution +path. Audience: developers reproducing issue #322 and evaluating Planner/control- +plane decisions. -Audience: developers reproducing issue #322 and the planner/control-plane evaluation. +The current replay guide is [o11y replay](../user_guide/o11y-replay.md). Benchmark +production and execution tooling belong to ASAPQuery-backend; planner-only +coverage and synthetic cached-result timing do not satisfy this evaluation. -## Scope +## Document map -Use offline sketch-bench CPU, elapsed time, state/memory, disk (when measured), -and errors against offline ground truth. No runtime ground truth, posterior -feedback, or self-estimated accuracy is required. Offline accuracy observations -do not establish formal guarantees on unseen distributions. +1. [Evaluation at a glance](#evaluation-at-a-glance) +2. [Worked example](#worked-example) +3. [Evidence contract](#evidence-contract) +4. [Execution and ownership](#execution-and-ownership) +5. [Comparison rules](#comparison-rules) +6. [Acceptance](#acceptance) -## Execution and ownership +## Evaluation at a glance -1. Pin isolated planner and control-plane worktrees from fetched main branches. - Preserve existing worktrees, including unresolved cost-model changes. -2. Evidence agent: implement the versioned artifact, validation, compatibility - matching, public cost-model integration, and fallback/decision tests. -3. Benchmark agent: run actual sketch-bench algorithms on uniform and Zipf - inputs, preserve raw output, export artifacts with source and environment - provenance, and document reproducible commands. -4. Replay agent: run the existing o11y PromQL corpus through the backend parser, - its ASAPPlanner call, and the backend typed binder (not a planner-only entry), - export binding/fallback coverage and planning latency, and separately identify - supplemental sketch workloads. -5. Integration owner: connect compatible evidence to the downstream control - plane, validate dependency compatibility, review other agents' changes, run - integration checks, and report evidence-supported comparisons. - -Steps 2–4 run concurrently after agreeing the artifact contract. Integration -uses their completed interfaces and measurements. Query-pattern changes are -limited to demonstrated blockers with regression coverage; unsupported query -semantics remain explicit in the report. +```text +sketch-bench measurements ──► versioned evidence artifact + │ +OpenMetrics workload ──► backend planning/binding ──► selected plan or fallback + │ + ▼ + decision and coverage report +``` -## Acceptance +Evidence includes measured CPU, elapsed time, state/memory, disk when measured, +and error against offline ground truth. It does not claim runtime ground truth, +posterior feedback or formal guarantees on unseen distributions. + +## Worked example + +Suppose sketch-bench measures KLL on one million Zipf-distributed values: + +```yaml +evidence: + algorithm: {kind: kll, k: 200} + dataset: {distribution: zipf, exponent: 1.1, events: 1000000} + environment: {cpu: example-x86, revision: abc123} + measurements: + update_cpu_ns: {mean: 48, samples: 20} + readout_cpu_ns: {mean: 2100, samples: 20} + serialized_state_bytes: {mean: 4096, samples: 20} + rank_error: {p99: 0.008, samples: 20} + observed_at: 2026-09-01T00:00:00Z +``` + +For a compatible o11y query, the backend binds this evidence to a concrete KLL +candidate and may rank it below exact execution. If the query uses a different +distribution, stale evidence or unsupported parameters, matching fails and the +planner retains its non-empirical choice or exact fallback. The report records +the match, selected plan and evidence provenance; it does not report missing +exact-baseline cost as zero. + +## Evidence contract + +Every artifact declares: -- Versioned schema and example; explicit units, algorithm/configuration, - dataset/distribution, environment, collection/validity times, sample count, - dispersion, and benchmark/model provenance. -- At least two actual sketch algorithms measured offline on uniform and skewed - inputs. CPU is distinct from elapsed time; serialization size is distinct - from heap memory and disk I/O. Unmeasured fields stay unavailable. -- Matching evidence affects a public planning cost/ranking/lifecycle boundary. - Tests cover changed decisions, missing/stale/mismatched evidence, invalid - values, and unchanged accuracy guarantees. -- The planner and control-plane evaluation preserve exact fallbacks and expose - unsupported shapes, measured provenance, and limits on benefit estimates. -- Reproducible measurement and replay commands, raw machine-readable results, - and a concise report distinguish actual measurements from modeled totals. +- schema version, units, algorithm and parameters; +- dataset, distribution and workload size; +- hardware/software environment and source revision; +- collection and validity times; +- sample count and dispersion; +- benchmark/model provenance; +- unavailable measurements explicitly. + +CPU and elapsed time are distinct. Serialized size, heap memory, disk I/O and +network bytes are distinct. Evidence affects a public cost, ranking or lifecycle +boundary only after compatibility and freshness validation. + +## Execution and ownership + +1. **Evidence producer:** run actual algorithms on uniform and Zipf inputs, + retain raw output and emit versioned artifacts. +2. **Evidence integration:** validate artifacts, match compatible records and + expose measured costs through the public cost-model boundary. +3. **Replay runner:** send the o11y PromQL corpus through the backend parser, + ASAPPlanner call and typed physical binder; record bindings and fallbacks. +4. **Integration owner:** pin revisions, run end-to-end checks and publish the + evidence-supported comparison. + +Artifact design, benchmarks and replay may proceed in parallel after agreeing on +the contract. Query-pattern changes require a demonstrated blocker and regression +coverage; unsupported semantics remain visible in the report. ## Comparison rules -Compare equivalent tasks, data, parameters, windows, horizons, and evaluation -cadences. Whole-plan estimates include build/update/readout, retained windows, -sharing, and raw residual work where evidence exists. Missing raw baseline or -physical evidence means an unavailable whole-plan speedup, not zero cost. -Microbenchmark algorithm comparisons are labeled separately. Disk usage and -network savings require their own measurements or explicit models. +Compare equivalent tasks, data, parameters, windows, horizons and evaluation +cadences. Whole-plan estimates include build/update/readout, retained state, +sharing and raw residual work where evidence exists. Algorithm microbenchmarks +remain separately labeled. + +Disk or network savings require measurements or an explicit model. Repeated-query +break-even requires compatible exact baseline, sketch maintenance and readout +measurements. Missing components make whole-plan speedup unavailable rather than +free. Offline estimates do not establish deployed end-to-end latency improvement. + +## Acceptance -Repeated-query break-even can be computed only when compatible exact baseline, -sketch build/maintenance, and readout measurements are present. It is an offline -estimate and does not establish deployed end-to-end latency improvement. +- At least two real sketch algorithms are measured on uniform and skewed inputs. +- Missing, stale, malformed and mismatched evidence cannot change a decision. +- Matching evidence changes at least one public planning decision in a test. +- Exact fallback and accuracy guarantees remain unchanged. +- Replay exposes supported bindings, unsupported shapes and provenance. +- Commands, raw machine-readable results and a concise report are reproducible. diff --git a/docs/design_docs/shape-aware-erp-v1.md b/docs/design_docs/shape-aware-erp-v1.md index 5d04d99af..8568589af 100644 --- a/docs/design_docs/shape-aware-erp-v1.md +++ b/docs/design_docs/shape-aware-erp-v1.md @@ -1,34 +1,94 @@ # Shape-aware ERP v1 -The backend obtains an observed shape from the live runtime-samples feedback -path and asks ASAPPlanner for the nearest compatible benchmark profile. A -runtime record carries `erp_observed_shape.observation` using Planner's shared -`ErpShapeObservation`: cardinality, observed event count, candidate fits and an -optional dataset fingerprint. The wrapper also reports burst ratio. The planning request -selects the ring via `observed_shape_source`. - -The observer retains both uniform and fitted Zipf hypotheses, with total-variation -distance against the observed rank masses and a sample-count-adjusted fit score. -The score is a heuristic fit quality, not a statistical confidence interval or -sketch-error guarantee. Unnamed and mixed distributions are not forced into one -family. Planner jointly matches admissible hypotheses against ERP records. -Custom data first matches its fingerprint when available, otherwise it can use -the same bounded shape matching as synthetic data. +Status: implemented planning contract. Audience: developers producing runtime +shape observations or matching Error–Resource Profile (ERP) evidence. + +## Document map + +1. [Design at a glance](#design-at-a-glance) +2. [Worked example](#worked-example) +3. [Observation and matching contract](#observation-and-matching-contract) +4. [Cost composition](#cost-composition) +5. [Miss and fallback behavior](#miss-and-fallback-behavior) +6. [Payload migration](#payload-migration) +7. [Population isolation](#population-isolation) + +## Design at a glance + +```text +runtime samples ──► bounded shape observer ──► ErpShapeObservation + │ +benchmark ERP records ─────────────────────────────┤ + ▼ + compatible evidence or miss + │ + ▼ + Planner cost and parameter choice +``` + +The observer retains uniform and fitted Zipf hypotheses. Planner matches +admissible hypotheses to compatible benchmark records. A fit score is a heuristic +measure of shape similarity, not a confidence interval or sketch-error guarantee. + +## Worked example + +```yaml +observation: + cardinality: 10000 + observed_events: 1000000 + empirical_fingerprint: null + fits: + - family: zipf + parameters: {exponent: 1.1} + goodness_of_fit: 0.03 + confidence: 0.92 + +match_policy: + max_goodness_of_fit: 0.05 + min_confidence: 0.90 + max_cardinality_ratio: 2.0 +``` + +If a fresh KLL benchmark record covers a compatible cardinality, event count, +parameters and Zipf fit, Planner uses its measured atomic update, merge and query +costs. A poorer fit, ambiguous hypothesis or excessive cardinality distance is a +miss. Custom data first matches an exact dataset fingerprint when one is present. + +For a five-minute query using one-minute panes, five query evaluations and one +shared materialization, the abstract operation counts are: + +```text +updates = observed updates +merges = 5 * (ceil(5m / 1m) - 1) = 20 +queries = 5 +state = retained panes * 1 +``` + +Planner multiplies those counts by compatible measured atomic costs; the runtime +observation does not directly claim total latency. + +## Observation and matching contract + +`erp_observed_shape.observation` uses Planner's shared `ErpShapeObservation` and +contains cardinality, observed event count, candidate fits and an optional dataset +fingerprint. The wrapper also records burst ratio; `observed_shape_source` +selects the observation ring. + +Each fit reports family, parameters, goodness of fit and confidence. Total- +variation distance is computed against observed rank masses with sample-count- +adjusted scoring. Unnamed or mixed distributions are not forced into one family. +Equal frequencies produce only the canonical uniform fit; Zipf exponent zero must +not create false ambiguity. Key count, key length and occupied interval count are bounded. Sparse interval -IDs do not allocate a dense vector. Any overflow or cap violation permanently -invalidates that observation window; callers must start a fresh observer rather -than publishing a biased partial snapshot. Profiles with too few benchmark -events, poor fit, ambiguous confidence, or excessive cardinality/parameter -distance are misses. +IDs do not allocate dense vectors. Overflow or a cap violation invalidates the +entire observation window; callers start a fresh observer instead of publishing +a biased partial snapshot. -On a hit, empirical parameters and measured atomic costs are used. On a miss, -malformed evidence, or drift, Hybrid mode retains the theoretical parameters; -if the runtime cannot deploy them or they exceed its memory limit, compilation -chooses exact execution. Empirical-only mode fails closed. +## Cost composition -For a pane width `p`, query window `w`, query executions `q`, updates `u`, -retained panes `r`, and shared materializations `m`, Planner composes: +For pane width `p`, query window `w`, query executions `q`, updates `u`, retained +panes `r` and shared materializations `m`: ```text updates = u * m @@ -38,40 +98,42 @@ state = r * m CPU = updates*Cupdate + merges*Cmerge + queries*Cquery ``` -This separates machine-specific atomic measurement from workload-specific -window planning and makes tumbling, sliding/pane, retention, and sharing costs -auditable. - -### Observation payload migration - -Runtime producers must replace the old single `shape` object with an -`observation` containing `cardinality`, `observed_events`, `fits`, and optional -`empirical_fingerprint`. Each fit supplies `family`, `parameters`, -`goodness_of_fit`, and `confidence`; shape-match policy must also supply the fit -quality and confidence thresholds. Old payloads are rejected rather than given -invented confidence. Publish a new observation after upgrading the producer. - -The bounded observer models ranked key frequencies. Its output does not describe -the numeric spacing of KLL sample values and must not be advertised as a general -numeric-distribution observation. Equal frequencies produce the canonical -uniform fit only: Zipf exponent zero describes the same distribution and must -not create a false ambiguity. Near-uniform, genuinely distinct fits still pass -through the normal ambiguity policy. - -### Population isolation in backend-local execution - -A temporal scalar summary has one state per source series when its selected -`SummaryAgg` uses `Reduction::PerEntity`. An explicit reduction with no grouping -keys has one pooled population. These are different materializations even when -source, sketch parameters, and the visible grouping-key list are identical. - -The compiler records shared `PopulationPartitioning` metadata in the runtime -configuration and DataDescriptor. Both identities include the partitioning; -installation checks it against the bound Planner DAG. Raw ingestion uses the -full source labels for per-entity routing and the configured grouping for pooled -routing. Memory estimates count per-entity states against source cardinality. -Legacy configurations without this metadata retain their existing routing rules. - -This is a source-isolation contract, not permission to skip a maintenance update -expression. Only already-supported scalar update expressions pass the compiler's -per-entity admission check; other subDAG updates still require a real evaluator. +This separates machine-specific atomic measurements from workload-specific +window, retention and sharing decisions. + +## Miss and fallback behavior + +Profiles with too few events, poor fit, ambiguous confidence, stale evidence or +excessive cardinality/parameter distance are misses. Malformed evidence also +fails matching. + +- **Hybrid mode:** retain theoretical parameters; choose exact execution if the + runtime cannot deploy them or they exceed its memory limit. +- **Empirical-only mode:** fail closed when compatible evidence is unavailable. + +## Payload migration + +Runtime producers replace the old single `shape` object with `observation`, +including `cardinality`, `observed_events`, `fits` and optional +`empirical_fingerprint`. Match policy supplies fit-quality and confidence +thresholds. Old payloads are rejected rather than assigned invented confidence. + +This ranked-frequency observation does not describe numeric spacing of KLL values +and must not be advertised as a general numeric-distribution model. + +## Population isolation + +`Reduction::PerEntity` creates one temporal scalar state per source series. An +explicit reduction with no grouping keys creates one pooled population. They are +different materializations even when source, parameters and visible grouping keys +look identical. + +The compiler stores `PopulationPartitioning` in runtime configuration and +`DataDescriptor`; both identities include it. Installation checks it against the +bound Planner DAG. Ingestion uses full source labels for per-entity routing and +configured grouping for pooled routing. Memory estimates charge per-entity state +against source cardinality. + +This isolation contract does not implement arbitrary maintenance expressions. +Only supported scalar update expressions pass per-entity admission; other subDAG +updates still require an evaluator.