Skip to content

Skill guidance for sampled pre-runs and their correctness traps #22

Description

@jaakla

Split out of #20 (workstream b). #20 keeps the deterministic CLI/marking work; this issue is the shipped-skill half.

Problem

Sampled pre-runs only help if the agent knows when to run one, how to choose the sample, and — most importantly — what a sampled run does and does not prove. Those are judgment calls with no deterministic oracle, so they belong in SKILL.md / references/, not in Python.

Guidance to add

When to sample. A checkable trigger derived from what the manifest already records — sources.*.access.file.row_count, sources.*.selection.completeness.matched, and AOI extent — rather than an unknowable "processing will take over 10 minutes". Above the threshold a sampled pre-run is required; below it, optional.

How to sample, in priority order. Sub-area (test AOI) is primary: it is the only strategy that keeps geometries that must intersect actually intersecting. Row limits, TABLESAMPLE, raster downsampling and geometry pre-simplification are fallbacks.

Named anti-patterns / correctness traps (the substance of this issue):

  • Clipping to a sample AOI breaks every neighbourhood operation at its boundary. Distance-to-nearest, buffers, catchments, and features crossing the cut are wrong by construction near the edge. A sample AOI needs a halo sized to the largest neighbourhood distance in the pipeline, and results inside the halo are not reportable. This is the same family as the existing "bbox overlap as scan gate, then exact predicate" rule.
  • Row-limit / TABLESAMPLE sampling destroys spatial coherence. A random 1% of parcels joined against a random 1% of roads yields approximately zero matches — so the moist run passes for the wrong reason, or fails for one. These strategies are smoke-test only, never result-shaped.
  • Raster downsampling changes results non-linearly — area, slope, hydrology, viewshed. Same conclusion.
  • Choosing a representative AOI is a judgment call. For global/continental data a naive sub-bbox lands in open ocean; sometimes the right sample is a dense city, sometimes deliberately outside one, sometimes genuinely offshore. State the selection rationale as an assumption.

The honest claim. A sampled run proves the pipeline executes: the manifest graph resolves, CRS handling holds, schemas line up, sources are reachable, the pagination proof works within the sample. It never proves the numbers. Counts and aggregates from a sampled run must not be surfaced as answers, written into validation.expectations, or shown in the dashboard as results.

Unaffected rules. Source pinning and the bounded-API completeness proof (numberMatched == returned) still apply within the sample — sampling is not a licence to skip pagination proof.

Acceptance criteria

  • Guidance lives in the right layer: routing/behaviour in SKILL.md, mechanics in references/processing.md (and references/data-sources.md where sampling interacts with pinning/completeness). SKILL.md does not grow disproportionately.
  • The traps above appear as universal anti-patterns, not as prose buried in a reference.
  • Per AGENTS.md, a material behaviour change needs eval coverage: a live eval case that a large-AOI task produces a sampled pre-run before the full run, and that its output is not presented as the validated answer. A fixture pass on Add partial run instructions and parameters for scalability #20's CLI work does not prove this half.

Depends on

#20 — the marking/status vocabulary the prose refers to has to exist first.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions