fix(query): bound the rollup's untracked allocation, and instrument the rest

[vishr]

Partial progress on #254. This no longer claims to fix the OOM — see below.

What this contains

quantile_cont → approx_quantile (verified, real, insufficient)

Holistic aggregates retain every value of every group in a vector on the raw allocator, outside anything memory_limit bounds.

30M rows at memory_limit='100MB': 861–912 MiB, with duckdb_memory() reporting 0.0 MiB. approx_quantile keeps a fixed-size t-digest per group: 8–9 MiB. Same substitution for PERCENTILE_CONT in the anomaly detector, which runs every 60s.

Deployed alone to the live demo host, this did not move the peak: 9.64 → 9.58 GiB. Real unbounded allocation, not the dominant one.

TestServiceRollupLatencyDoesNotAllocatePerRow gates it, verified in both directions: 912 MiB fail on the old expression, 9 MiB pass on the new.

A per-tag fanout_duckdb_memory_bytes gauge

So process_resident_memory_bytes − Σ fanout_duckdb_memory_bytes − go_memstats_heap_sys_bytes is the untracked heap directly. The absence of this series is why several hypotheses were chased on inference before anyone scraped the endpoint.

What I removed from this PR, and why

An earlier revision dropped union_by_name=true from the Parquet views. That was wrong and review caught it.

It worked, spectacularly — peak 9.64 → 4.49 GiB on the live host, flat while batch count grew 1,744 → 6,263. Bind opens every file in the glob and holds a reader and footer per file for the query's life; measured at 510 MiB vs 89 MiB per query on 3,281 live files, with duckdb_memory() reporting an identical 69.0 MiB either way.

But verified on DuckDB 1.5.5:

no union_by_name:   Invalid Input Error: schema mismatch in glob: column "newcol" was
                    read from the original file a.parquet, but could not be found in b.parquet
union_by_name=true: id=1 newcol=x / id=2 newcol=<null>

ensureSchemaBatch writes an empty _schema.batch from the current binary's structs precisely so the glob carries the full column set, and CreateViews names every column explicitly and binds eagerly. So a build that adds a column would fail at NewDuck and the process would refuse to start until every pre-deploy batch aged out.

My claimed mitigation was also wrong: selectCompactionBatches only merges ≥2 batches sharing a day and generation, so old-schema files can survive to retention.

That trades an OOM kill for a process that won't boot. Worse. Reverted.

So the OOM is not fixed

The dominant term is real and measured — per-file bind cost × concurrent binders × live file count. The flag that causes it is load-bearing and cannot simply be removed.

The bounded fix is to bound the file count. Compaction is losing badly to ingest: 1,744 → 6,263 live batches in 38 minutes on the demo. At ~128 KB/file of bind cost, holding the batch count near a few hundred keeps per-query cost in the tens of MB and makes the ceiling memory_limit plus a fixed margin. That's a separate change: raise compaction throughput, or add ingest backpressure.

Review findings addressed

• RefreshDuckDBMemory had zero callers — the metric family exported nothing, indistinguishable from "DuckDB holds nothing". Now a pull-based collector.
• releaseMemoryGauges was assigned but never invoked; Close now releases it, so a closure over a closed pool can't freeze the gauges at stale values.
• Reset() before republish, so a tag that stops appearing stops being reported.
• rows.Err() checked rather than publishing a partial sum as truth.
• Doc comment no longer displaces SetDuckDBPoolSource's.
• Detector comment moved onto the query it describes.
• RSS read from /proc/self/statm where available, so the gate doesn't depend on a ps supporting -o/-p; long scan skipped under -short.

Verified and not changed: DuckDB dedupes the two identical approx_quantile expressions — EXPLAIN shows two, not four.

just check passes.