SketchStore query-time candidate lookup key doesn't match registration key under `grouping` config — confirmed-present sketch data returns CapabilityMiss

[zzylol]

Summary

Per docs/data_model.md (ASAPCollector-public), a series' identity (series_id/sid) should be fully determined by agg_id + metric + labels — nothing else, and in particular not by which collector/replica produced a given envelope. A recent real-deployment benchmark session independently confirmed this part of the design is correct in practice: 4 separate asap-otel collectors, each running a whole_stream-mode DDSketch aggregation over the same metric, correctly merged into a single backend sid (SketchStore: 1 instance, 1 sid) — exactly as the data model predicts, since nothing about collector identity is (or should be) part of sid identity.

However, in a separate real-deployment debugging session earlier in the same benchmarking effort, we found a query-time bug that appears to violate the same data model: a query against confirmed-present, correctly-ingested sketch data returned CapabilityMiss (empty result) — reproducing even at the smallest possible scale (1 collector, 200 series, a single fully-qualified point-lookup needing no merge at all).

Repro (as run)

• Single asap-otel collector, data_plane backend, grouping: [series_id] in the streaming config, KLL sketch family, 200 series.
• Confirmed via [MEMORY_DIAG] logging: SketchStore genuinely held 200 correctly-ingested KLL sketch instances (177.86KB real payload).
• Both a fully-qualified single-series query (quantile_over_time(0.99, bench_latency_ms{series_id="s000000"}[30s])) and a cross-series query (quantile(0.99, bench_latency_ms)) returned CapabilityMiss.

Diagnostic trail (RUST_LOG=debug)

The query correctly reaches engine.rs::execute(); the control-plane analyzer (analyze_promql_for_asap_tier) accepts the query shape (not the Branch-1 "unsupported" path); it builds a KeyByLabelNames with 0 labels for both query shapes; it then hits the code path at data_plane/src/query_engines/asap_query_engine/engine.rs:790 whose own comment describes exactly this situation — "candidates populated, but instances_matching returns empty/Ghost" — and returns CapabilityMiss.

This points at a mismatch between:

• how the PromQL analyzer (asap_tier_analysis.rs) computes a candidate's expected SketchStore lookup key for a given query, vs.
• how SketchStoreSink actually registers stored instances under a grouping: [series_id] streaming config.

The failure signature was identical for both the single-series point-lookup and the cross-series query, which suggests this is a general SketchStore candidate-key-computation gap rather than something specific to cross-series merging (the cross-series/merge path itself looks architecturally sound and collector-identity-agnostic by design — see the whole_stream confirmation above and SketchStore::instances_matching's implementation, which has no collector/agent field anywhere in its index, matching docs/data_model.md's intent).

Note: the affected code lives in ProjectASAP/ASAPQuery-backend (data_plane/src/query_engines/asap_query_engine/engine.rs, asap_tier_analysis.rs, SketchStoreSink), not in this repo — filing here per the reporting convention used elsewhere in this benchmarking effort's tracked design docs; please cross-file/move if there's a preferred tracker for backend-side bugs.

Not yet done

Tracing into asap_tier_analysis.rs/instances_matching far enough to pin the exact key-computation divergence — this issue is filed at the "confirmed real, narrowed to two specific files, not yet root-caused to a line" stage, not a fix.

Suggested next step

Add a debug/trace log at the point instances_matching is called from find_candidates, printing both the analyzer's computed candidate key and the actual registered keys in the index for the metric in question, to see exactly which field(s) diverge (label set shape? agg_id scoping? grouping-key normalization?).