Skip to content

Non-interactive cost estimation and the scale escalation ladder #23

Description

@jaakla

Split out of #20 (workstream c).

Problem

#20 asks that very large analyses "try to estimate final runtime and ask user for confirmation". Two things block that as originally stated:

  1. The CLI cannot block on stdin. openmapstack run is invoked non-interactively by subprocess.run in openmapstack/cli.py and by the clean-rerun protocol in openmapstack/rerun.py, and by CI. An interactive confirmation prompt would hang the harness and every eval that shells out to it.
  2. An a-priori runtime estimate has no basis. There is no cost model. The only credible estimate is sampled wall-clock x scale factor, which requires Add partial run instructions and parameters for scalability #20's sampled run to have already recorded its scale factor and per-step timings.

Proposed split of responsibility

  • CLI: emits a machine-readable projection and exits. Something like openmapstack run project.yaml --estimate --json, reporting projected wall-clock, projected output size, the scale factor it extrapolated from, and which sampled run record it derived them from. Never prompts.
  • Agent/user layer: owns the confirmation. The skill decides whether to ask before committing to a multi-hour job.

Escalation ladder

When a job projects as very large, offer intermediate rungs rather than a binary accept/reject — city (typically the first test run), county, small country, large country, continent, global. Each rung is an ordinary sampled run from #20 with a larger AOI, so the ladder is a policy over an existing mechanism, not new machinery.

Acceptance criteria

  • --estimate is non-interactive, machine-readable, and honest about its provenance: it must say it extrapolated from a specific sampled run, and refuse to guess when no sampled run exists rather than inventing a number.
  • The escalation ladder is documented as skill guidance with the confirmation happening agent-side.
  • Estimation error is reported as a range, not a false-precision point value.
  • Tests cover: estimate with a prior sampled run, estimate with no sampled run (refuses, does not fabricate), and that no code path reads stdin.

Depends on

#20 — the sampled run record carrying scale factor and step timings is the input this consumes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions