You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Split out of #20 (workstream b). #20 keeps the deterministic CLI/marking work; this issue is the shipped-skill half.
Problem
Sampled pre-runs only help if the agent knows when to run one, how to choose the sample, and — most importantly — what a sampled run does and does not prove. Those are judgment calls with no deterministic oracle, so they belong in SKILL.md / references/, not in Python.
Guidance to add
When to sample. A checkable trigger derived from what the manifest already records — sources.*.access.file.row_count, sources.*.selection.completeness.matched, and AOI extent — rather than an unknowable "processing will take over 10 minutes". Above the threshold a sampled pre-run is required; below it, optional.
How to sample, in priority order. Sub-area (test AOI) is primary: it is the only strategy that keeps geometries that must intersect actually intersecting. Row limits, TABLESAMPLE, raster downsampling and geometry pre-simplification are fallbacks.
Named anti-patterns / correctness traps (the substance of this issue):
Clipping to a sample AOI breaks every neighbourhood operation at its boundary. Distance-to-nearest, buffers, catchments, and features crossing the cut are wrong by construction near the edge. A sample AOI needs a halo sized to the largest neighbourhood distance in the pipeline, and results inside the halo are not reportable. This is the same family as the existing "bbox overlap as scan gate, then exact predicate" rule.
Row-limit / TABLESAMPLE sampling destroys spatial coherence. A random 1% of parcels joined against a random 1% of roads yields approximately zero matches — so the moist run passes for the wrong reason, or fails for one. These strategies are smoke-test only, never result-shaped.
Choosing a representative AOI is a judgment call. For global/continental data a naive sub-bbox lands in open ocean; sometimes the right sample is a dense city, sometimes deliberately outside one, sometimes genuinely offshore. State the selection rationale as an assumption.
The honest claim. A sampled run proves the pipeline executes: the manifest graph resolves, CRS handling holds, schemas line up, sources are reachable, the pagination proof works within the sample. It never proves the numbers. Counts and aggregates from a sampled run must not be surfaced as answers, written into validation.expectations, or shown in the dashboard as results.
Unaffected rules. Source pinning and the bounded-API completeness proof (numberMatched == returned) still apply within the sample — sampling is not a licence to skip pagination proof.
Acceptance criteria
Guidance lives in the right layer: routing/behaviour in SKILL.md, mechanics in references/processing.md (and references/data-sources.md where sampling interacts with pinning/completeness). SKILL.md does not grow disproportionately.
The traps above appear as universal anti-patterns, not as prose buried in a reference.
Per AGENTS.md, a material behaviour change needs eval coverage: a live eval case that a large-AOI task produces a sampled pre-run before the full run, and that its output is not presented as the validated answer. A fixture pass on Add partial run instructions and parameters for scalability #20's CLI work does not prove this half.
Depends on
#20 — the marking/status vocabulary the prose refers to has to exist first.
Split out of #20 (workstream b). #20 keeps the deterministic CLI/marking work; this issue is the shipped-skill half.
Problem
Sampled pre-runs only help if the agent knows when to run one, how to choose the sample, and — most importantly — what a sampled run does and does not prove. Those are judgment calls with no deterministic oracle, so they belong in
SKILL.md/references/, not in Python.Guidance to add
When to sample. A checkable trigger derived from what the manifest already records —
sources.*.access.file.row_count,sources.*.selection.completeness.matched, and AOI extent — rather than an unknowable "processing will take over 10 minutes". Above the threshold a sampled pre-run is required; below it, optional.How to sample, in priority order. Sub-area (test AOI) is primary: it is the only strategy that keeps geometries that must intersect actually intersecting. Row limits,
TABLESAMPLE, raster downsampling and geometry pre-simplification are fallbacks.Named anti-patterns / correctness traps (the substance of this issue):
TABLESAMPLEsampling destroys spatial coherence. A random 1% of parcels joined against a random 1% of roads yields approximately zero matches — so the moist run passes for the wrong reason, or fails for one. These strategies are smoke-test only, never result-shaped.The honest claim. A sampled run proves the pipeline executes: the manifest graph resolves, CRS handling holds, schemas line up, sources are reachable, the pagination proof works within the sample. It never proves the numbers. Counts and aggregates from a sampled run must not be surfaced as answers, written into
validation.expectations, or shown in the dashboard as results.Unaffected rules. Source pinning and the bounded-API completeness proof (
numberMatched == returned) still apply within the sample — sampling is not a licence to skip pagination proof.Acceptance criteria
SKILL.md, mechanics inreferences/processing.md(andreferences/data-sources.mdwhere sampling interacts with pinning/completeness).SKILL.mddoes not grow disproportionately.Depends on
#20 — the marking/status vocabulary the prose refers to has to exist first.