agentcast

Forecast an AI coding agent's wall-clock, cost, and probability of success — before it runs — from just the prompt, the repo, and the model.

Teams are handing real work to coding agents (Claude Code, Codex, …) and have no idea what to expect. agentcast learns from your own Claude Code transcripts and answers the question directly: "If I give the agent this task, how long will it take, what will it cost, and will it actually work?"

It's a sibling to buildcast — buildcast forecasts remaining work from git-commit velocity (looking at history); agentcast forecasts a task you haven't run yet.

Your data never leaves your machine. agentcast trains locally on your own ~/.claude transcripts. This repo ships code only — no prompts, no dataset, no trained model.

---

What it predicts

Output	How good	Method
Wall-clock P50 / P85 / P95	CRPS skill +4.4% vs baseline · coverage calibrated to 50 / 85 / 95	quantile regression + conformal recalibration
Cost (USD) P50 / P85 / P95	CRPS skill +8.1% · coverage 50 / 85 / 95	cache-aware token pricing
P(success)	AUC 0.78 seen-repo · 0.69 cold-start · Brier skill +24%	calibrated classifier, honest dual-CV

Every forecast uses only ex-ante features — what you know before the run: prompt text, repo, model, repo content (has-tests / language / size / CI), and position in the session. Run outcomes (tokens, turns, whether tests passed) are used only to build training labels — never as inputs. No leakage.

---

The honest part

Success is reported two ways, because they answer different questions:

• Seen-repo (AUC 0.78) — you've run the agent in this repo before. Its history is genuinely predictive. This is the realistic "forecast my team's ongoing work" case.
• Cold-start (AUC 0.69) — a repo the model has never seen (repo-held-out CV). Needs real task understanding; harder.

The gap between them is exactly how much the model memorizes per-repo base rates vs generalizes. Most tools never show you this. agentcast bakes both into every training run so it can't oversell itself.

What didn't work (proven, not assumed)

Half of rigor is knowing what to throw away:

Lever	Result
LLM "difficulty" scoring of each prompt	~0 gain (an early 0.89 AUC was a label-leakage artifact — caught and killed)
More training data	success learning curve flat after 50% — not data-bound
Conversation-context features	+0.004 — noise
Prompt text alone	AUC ≈ 0.59 — wording barely predicts success

The one cold-start lever that did work: generalizable repo-content features (has-tests / language / size / CI), which transfer to unseen repos — unlike repo identity. That's the 0.64 → 0.69 lift.

Why it ceilings here

Three independent signals — flat learning curve, weak text, repo-CV drop — say the residual is irreducible: whether a task succeeds depends on codebase state and ambiguity that don't exist in the prompt yet. Published agent-success models reach ~0.84+, but using the agent's mid-run response; agentcast predicts purely ex-ante, a harder and different problem. Software effort estimation broadly tops out near ±50–100% MAPE — the win is calibration, not point accuracy. All numbers are in-distribution (one author's history) with ~±0.03 CV noise; run it on your own data to see how it transfers.

---

How it works

~/.claude/**/*.jsonl
   │
   ├─ extract.py    sessions → one row per task: prompt features, repo, model,
   │                wall-clock (idle-gap aware), cache-aware cost, token usage
   │
   ├─ outcome.py    mine tool-results per task: did edits apply? did a test/build/
   │                lint command exit clean? → objective success signal
   │
   └─ run.py train  • LightGBM quantile regression in log space → calibrated
                      P50/P85/P95 time & cost (conformal recalibration)
                    • HistGBM + sigmoid calibration → P(success), dual-CV reported

Quickstart (on your own transcripts)

pip install -r requirements.txt

python extract.py     # ~/.claude transcripts → data/tasks.csv      (local only)
python outcome.py     # objective tool-outcome labels → data/outcomes.csv
python run.py train   # backtest receipt + fit + save model/agentcast.pkl

# forecast a task you haven't run yet:
python run.py predict "add retry logic to the API client and write tests" /path/to/repo claude-opus-4-8

# or forecast the latest prompt in your most recent session:
python run.py live

  "add retry logic to the API client and write tests"
  repo /path/to/repo · model claude-opus-4-8

  WALL-CLOCK   P50   3m   P85   12m   P95   17m
  COST         P50 $1.63  P85 $13.60  P95 $29.07
  P(SUCCESS)   71%   looks tractable

---

Method notes

• Quantile forecasting — one LightGBM per quantile on log1p(target), exp-back and sorted so quantiles never cross. Scored with CRPS (≈ mean pinball loss over the quantile grid).
• Calibration — per-quantile conformal recalibration from pooled out-of-fold ratios, so the P85 band actually covers ~85% of outcomes.
• Objective success label — a test/build/lint command exiting clean, or a commit/PR landing = success; interrupted or an explicit "no" = fail; the ambiguous middle is left unlabeled (that noise was the old ceiling).
• No leakage — GroupKFold by session, and by repo for the cold-start number. The featurizer is fit only on training folds.

Research provenance

The investigation behind the numbers is reproducible once you've run extract.py + outcome.py on your own data:

• push_success.py — learning curve + context-feature ablation
• label_test.py — label-definition comparison
• probe_c.py — is the gain real signal, or just task-shape?
• repo_cv.py — session-held-out vs repo-held-out (the memorization test)
• repo_content.py — the cold-start repo-content lever
• difficulty.py — the LLM-difficulty dead-end, kept as a documented negative result (optional; needs ANTHROPIC_API_KEY)

Privacy

data/ (your prompts) and model/ (which encodes them) are git-ignored and never committed — everything runs locally. The only optional network call is difficulty.py, which is off the main path.

License

MIT © Ishaan Pandey