diff --git a/CodeyBox.slnx b/CodeyBox.slnx index ababf0e0..e52845f8 100644 --- a/CodeyBox.slnx +++ b/CodeyBox.slnx @@ -10,6 +10,7 @@ + diff --git a/docs/concepts/agents.md b/docs/concepts/agents.md index 77636f43..9ebb7c4c 100644 --- a/docs/concepts/agents.md +++ b/docs/concepts/agents.md @@ -18,6 +18,7 @@ tooling, not in the agent runner contract. | `opencode` | `opencode` | `OPENCODE_AUTH_JSON`, written to `~/.local/share/opencode/auth.json` | `CODEYBOX_OPENCODE_AUTH_FILE` | | `antigravity` | `agy` | `CODEYBOX_ANTIGRAVITY_OAUTH_CREDS_JSON` (OAuth bundle, written to `~/.gemini/antigravity-cli/antigravity-oauth-token`) | `CODEYBOX_ANTIGRAVITY_OAUTH_CREDS_JSON` | | `crock` | `crock` | `CROCK_CONFIG_JSON` (file-materialised to `~/.crockcode/config.json`) | `CODEYBOX_CROCK_CONFIG_JSON` | +| `pi` | `pi` | `ANTHROPIC_API_KEY` (provider API key; other providers use their own variable from pi's provider table — see [Pi quirks](../reference/agent-quirks.md#pi-coding-agent-pi)) | `CODEYBOX_PI_API_KEY` | The sandbox-side env name is what the agent CLI reads. The host-side name is what the orchestrator looks up when building the credential bundle — for most @@ -49,6 +50,7 @@ the most common cause of fresh-class dispatch failures. | `cursor` | `curl -fsSL https://cursor.com/install \| bash` | Installs as `agent` (not `cursor-agent`). See [Cursor quirks](../reference/agent-quirks.md#cursor-cli-agent). | | `opencode` | `curl -fsSL https://opencode.ai/install \| bash` | Plaintext stdout only — no structured stream. | | `antigravity` | *operator-supplied — stage the `agy` binary on the host and ship it via `CodeyBox:MultipassExecutableProvisions` or `CodeyBox:Incus:ExecutableProvisions`, matching the selected provider* (see [Antigravity quirks](../reference/agent-quirks.md#google-antigravity-cli-agy)). Do not use `curl -fsSL https://antigravity.google/cli/install.sh \| bash`: that URL serves the landing page, not a script, and piping HTML into `bash` fails silently when the runcmd ends with `\|\| true`. | Installs the proprietary `agy` CLI on the non-login sandbox PATH. Multi-model gateway — each gateway model id is a separate quota bucket. Configure each accepted model as its own `AgentClass` member; the router gates per-model via the existing `(AgentKind, ModelId)` exhaustion key. | +| `pi` | `npm install -g --ignore-scripts @earendil-works/pi-coding-agent` | MIT-licensed; needs Node.js on the image. `--ignore-scripts` skips npm lifecycle scripts during install. See [Pi quirks](../reference/agent-quirks.md#pi-coding-agent-pi). | Verify each command against its upstream install docs at the time of baking — versions and install URLs change. Multipass and Incus keep independent bake @@ -168,6 +170,7 @@ credentials before they waste expensive compute. | `copilot` | *(no probe)* — always passes | — | | `cursor` | *(no HTTP probe — Cursor exposes no public usage endpoint)* — verifies the credential bundle carries `CODEYBOX_CURSOR_AUTH_JSON`; real auth check happens on first CLI call | — | | `opencode` | *(no network call)* — credential-presence check only | `OPENCODE_AUTH_JSON` | +| `pi` | *(no network call — pi fronts 30+ providers, so no single endpoint validates the credential)* — verifies the bundle carries `ANTHROPIC_API_KEY`; real auth check happens on first CLI call | `ANTHROPIC_API_KEY` | Each probe sends the minimal possible request (`max_tokens=1`). A 2xx response means the credential is valid. 401/403 is classified as `"auth"` failure. diff --git a/docs/operating/costs.md b/docs/operating/costs.md index 187662df..1af82626 100644 --- a/docs/operating/costs.md +++ b/docs/operating/costs.md @@ -90,6 +90,7 @@ events. | Agent | Bundled rates | Why | |---|---|---| | `claude`, `codex`, `gemini` | yes | the provider publishes per-token list prices | +| `pi` | yes, for the shipped Anthropic-backed member | pi is a multi-provider front; the bundled bucket mirrors Anthropic on-demand rates keyed by the bare `message.model` id. Operators fronting other providers add that provider's list prices under `CodeyBox:AgentPricing`. | | `opencode` | yes, estimated | OpenCode Go is subscription-priced. Bundled rates are a single subscription-equivalent USD/M per model (same value for input, cached, and output), derived from the $12/5h budget, each model's requests-per-5h limit, and the token mix documented at [opencode.ai/docs/go](https://opencode.ai/docs/go). Keys are `opencode-go/`. | | `cursor`, `copilot` | no | flat-rate subscriptions with no published per-token price | diff --git a/docs/reference/agent-quirks.md b/docs/reference/agent-quirks.md index d5e039e9..3bad6ba7 100644 --- a/docs/reference/agent-quirks.md +++ b/docs/reference/agent-quirks.md @@ -669,3 +669,85 @@ root, so a misconfiguration fails as an Infrastructure error rather than a catastrophic host mount. Only sandbox providers that preserve a live local Unix socket support this fallback. The daemon owns the tunnel + MCP tools and, if configured with its own key, is what bills the batch. + +### Pi coding agent (`pi`) + +**Install in the sandbox image** — add the install line to +`CodeyBox:MultipassExtraRuncmd` or `CodeyBox:Incus:ExtraRuncmd`, matching the +selected provider (verified against pi 0.85.1, 2026-09-14): + +```sh +npm install -g --ignore-scripts @earendil-works/pi-coding-agent +``` + +MIT-licensed ([repo](https://github.com/badlogic/pi-mono/tree/main/packages/coding-agent), +npm `@earendil-works/pi-coding-agent`). Lean terminal agent (Read/Write/Edit/Bash +tools) fronting 30+ providers through its pi-ai layer. + +**Non-interactive invocation.** The runner drives `pi --mode json` with the +prompt on stdin and NO positional prompt argument: + +```sh +pi --mode json --no-session --offline [--model ] [--thinking ] +``` + +`--mode json` was chosen over raw `-p` and `--mode rpc`: `-p` prints only the +final response text (usage, dispatch model, and terminal error shape would be +unrecoverable), while `--mode rpc` is a bidirectional prompt/response protocol +needing a driver loop for no extra signal on a one-shot run. `--mode json` +exits after the run like `-p` but emits one JSON event per stdout line, with +cumulative `usage {input, output, cacheRead, cacheWrite, totalTokens}` and +`model` on the assistant message frames. `--no-session` skips persisting +`~/.pi/agent/sessions` (the VM is ephemeral); `--offline` disables pi.dev +startup network (version checks, telemetry). The runner deliberately passes +NEITHER `--approve` NOR `--no-approve`: the sandbox tree is untrusted, so the +`ask` default (ignore project resources) is the safe posture. + +**Exit-zero errors.** Pi exits 0 even when the run dies before producing +output (verified: missing API key and a provider 401 both exit 0 with the +cause only in the event stream). The runner lifts the terminal error +(`stopReason: "error"` + `errorMessage` on the message frames, or the +plaintext `No API key found for the selected model.`) into +`TerminalDiagnostic`, so the pipeline's no-changes branch parks quota/auth +give-ups instead of dead-lettering them as "produced no changes" — the same +shape `agy` has. + +**Authentication.** Provider API keys from the environment (`ANTHROPIC_API_KEY`, +`OPENAI_API_KEY`, `GEMINI_API_KEY`, … — full table in pi's `providers.md`). +The shipped credential mapping wires host `CODEYBOX_PI_API_KEY` to +sandbox-side `ANTHROPIC_API_KEY`; operators fronting other providers add that +provider's variable to the mapping. Interactive `/login` state is not shipped +into sandboxes. Prefer `provider/id`-qualified `ModelId` values +(e.g. `anthropic/claude-haiku-4-5`): pi's default provider is google, so a bare +id can resolve against the wrong catalog. + +**Reasoning effort.** `ReasoningMode` maps 1:1 onto `pi --thinking` +(`off|minimal|low|medium|high|xhigh|max`). Only exact allowlist members are +emitted; anything else is ignored rather than passed through to fail the CLI. + +**Quota probe.** Ships as Unknown-only: pi has no meterable quota endpoint +(it fronts 30+ providers). The router's `QuotaUnknownPolicy` (default +`UseObservedFailures`) gates dispatch via observed failure history, and +`PiQuotaFailureDetector` classifies the relayed provider errors (401/auth +shapes → Unauthorized; 429/rate-limit → RateLimitExceeded; 402/billing → +LimitReached) with operator-extensible rows under +`CodeyBox:QuotaFailurePatterns:pi`. + +**Smoke probes.** Host-side `PiSmokeProbe` is a credential-presence check only +(no network call — no single endpoint validates a multi-provider credential). +`PiInVmSmokeProbe` execs `pi --version` plus a `pi --help | grep -q -- --mode` +assertion, so a pi build that dropped the JSON event stream benches at smoke +time instead of failing first dispatch. + +**Model-list probe.** `pi --list-models` needs an authenticated provider plus +network, so the host-side probe returns the curated `PiKnownModels` seed +instead of live-reading the catalog. Operator `ModelId` values absent from the +seed surface as a startup warning, never a hard reject (pi accepts fuzzy and +provider-qualified ids beyond the seed). + +**Cost attribution.** `PiCostExtractor` takes the latest (cumulative) `usage` +frame and the bare `message.model` id (pi strips the `provider/` qualifier in +`message.model`). Bundled rates live in `agent-pricing-defaults.json` under the +`pi` bucket for the shipped Anthropic-backed member; operators fronting other +providers add that provider's list prices there (or under +`CodeyBox:AgentPricing`) keyed by the bare model id. diff --git a/src/CodeyBox.Agents.Pi/CodeyBox.Agents.Pi.csproj b/src/CodeyBox.Agents.Pi/CodeyBox.Agents.Pi.csproj new file mode 100644 index 00000000..ff8aae0c --- /dev/null +++ b/src/CodeyBox.Agents.Pi/CodeyBox.Agents.Pi.csproj @@ -0,0 +1,19 @@ + + + + + + + + + + net10.0 + enable + enable + + + + + + + diff --git a/src/CodeyBox.Agents.Pi/PiAgentRunner.cs b/src/CodeyBox.Agents.Pi/PiAgentRunner.cs new file mode 100644 index 00000000..510afd54 --- /dev/null +++ b/src/CodeyBox.Agents.Pi/PiAgentRunner.cs @@ -0,0 +1,261 @@ +using System.Text.Json; +using CodeyBox.Agents; +using CodeyBox.Core; +using CodeyBox.Sandbox; + +namespace CodeyBox.Agents.Pi; + +/// +/// Drives the Pi coding-agent CLI (binary pi, npm +/// @earendil-works/pi-coding-agent, MIT-licensed) in non-interactive +/// mode via its --mode json transport: one JSON event per stdout line, +/// with cumulative provider-reported usage on message events and a +/// terminal agent_end frame. +/// +/// Transport decision (verified against pi 0.85.1). +/// --mode json is the primary transport — not raw -p and not +/// --mode rpc. Raw -p prints only the final response text, so +/// token usage, the dispatch model id, and the terminal error shape would all +/// be unrecoverable. --mode rpc is a bidirectional prompt/response +/// protocol (prompt command in, events out) that needs a driver loop for no +/// additional signal on a one-shot run. --mode json exits after the run +/// like -p but keeps every event we need: message_end / +/// turn_end / agent_end carry message.usage {input, output, +/// cacheRead, cacheWrite, totalTokens}, message.model, and — on +/// failure — stopReason: "error" with errorMessage. +/// +/// Exit-zero errors. Pi exits 0 even when the run dies before +/// producing output (verified: missing API key and a 401 both exit 0 with the +/// cause only in the event stream). therefore lifts the +/// terminal error region into via +/// so the pipeline's no-changes branch can +/// park quota/auth failures instead of dead-lettering them as "produced no +/// changes" — the same give-up shape agy has (see +/// AgentResult.TerminalDiagnostic). +/// +/// Auth. Pi reads provider API keys from the environment +/// (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, … — +/// full table in pi's providers.md). The shipped credential mapping wires the +/// host CODEYBOX_PI_API_KEY to sandbox-side ANTHROPIC_API_KEY; +/// operators fronting other providers extend the mapping with that provider's +/// variable from the same table. Subscription (/login) state is +/// interactive-only and is not shipped into sandboxes. +/// +/// Project trust. Non-interactive pi loads AGENTS.md context +/// files but ignores project-local settings/extensions/skills unless the +/// project is trusted. This runner deliberately passes NEITHER +/// --approve NOR --no-approve: the sandbox working tree is +/// untrusted repo content, so the ask default (ignore project +/// resources) is the safe posture. An operator that vets their tree can set a +/// saved trust decision on the image; the runner must not override that. +/// +public sealed class PiAgentRunner : CliAgentRunnerBase, IStructuredStreamAgentRunner, IAgentDefaultModelProvider, ITextOnlyAgentRunner +{ + private readonly AgentDefaultsSnapshot? _defaults; + + public PiAgentRunner() : this(defaults: null) { } + + /// + /// Live snapshot of per-agent default model IDs (see + /// ). Supplies + /// when a caller does not pass an explicit + /// model, so the dispatch model is sourced from hot-reloadable config + /// rather than a hardcoded literal. + /// + public PiAgentRunner(AgentDefaultsSnapshot? defaults) + { + _defaults = defaults; + } + + public override AgentKind Kind => AgentKind.Pi; + + /// + /// Default pi binary name inside the sandbox. Shared with + /// PiInVmSmokeProbe so the smoke check and the real runner always + /// invoke the same binary. + /// + public const string DefaultBinary = "pi"; + + /// Path to the pi binary inside the sandbox. Defaults to . + public string Binary { get; init; } = DefaultBinary; + + /// + /// Default model passed to --model when the agent-class member + /// does not override it. Sourced live from + /// (config key CodeyBox:AgentDefaults[pi]). Pi accepts + /// provider/id-qualified ids, fuzzy patterns, and an optional + /// :thinking suffix — prefer a qualified id so the run does not + /// depend on pi's own startup default. + /// + public string? DefaultModelId => _defaults?.GetDefault(Kind.Value); + + /// + /// The thinking levels pi --help accepts for --thinking + /// (verified against pi 0.85.1). only emits + /// the flag for an exact (case-insensitive) member of this set; anything + /// else is ignored so a typo cannot fail a dispatch at the CLI layer. + /// + internal static readonly IReadOnlySet ThinkingLevels = new HashSet(StringComparer.OrdinalIgnoreCase) + { + "off", "minimal", "low", "medium", "high", "xhigh", "max", + }; + + protected override IReadOnlyList DirectCredentialEnvironmentVariables => ["ANTHROPIC_API_KEY"]; + + protected override string PreemptProcessPattern => Binary; + + /// + /// Verifies --mode json support with pi --help. The runner's + /// only transport is the JSON event stream, so a binary that no longer + /// advertises the flag must fail closed here rather than dispatch into an + /// unparseable plaintext run. + /// + public async Task SupportsStructuredStreamAsync(ISandbox sandbox, CancellationToken ct = default) + { + var help = await sandbox.ExecAsync(new SandboxExec + { + Argv = [Binary, "--help"], + }, ct).ConfigureAwait(false); + + if (!help.Success) + return false; + + var output = string.Concat(help.Stdout, "\n", help.Stderr); + return output.Contains("--mode", StringComparison.Ordinal) + && output.Contains("json", StringComparison.Ordinal); + } + + protected override AgentInvocation BuildInvocation( + string prompt, + AgentCredential? credential, + string? modelId = null, + string? reasoningMode = null, + bool captureStructuredStream = false) + { + // `pi --mode json` processes the prompt and exits (like -p) while + // emitting the full session event stream on stdout. It is the ONLY + // transport this runner speaks — even when the caller did not ask for + // structured capture — so cost attribution (usage on message events), + // failure classification (stopReason/errorMessage), and stream parsing + // never depend on which call path dispatched the run. --mode rpc is + // rejected deliberately: it needs a bidirectional driver loop for no + // extra signal on a one-shot run. + var argv = new List { Binary, "--mode", "json" }; + + // Ephemeral session: the sandbox VM is discarded after the run, so + // persisting ~/.pi/agent/sessions buys nothing and leaves unbounded + // session files behind on long-lived images. + argv.Add("--no-session"); + + // Disable pi.dev startup network (version check, package update + // checks, install telemetry). The sandbox network allow-list does not + // include pi.dev, and stalling the run on it serves no dispatch + // purpose. Equivalent to PI_OFFLINE=1; the flag keeps the posture + // visible in argv rather than hidden in the environment. + argv.Add("--offline"); + + // Fall back to the config-sourced default when the caller passes no + // explicit model, mirroring GeminiAgentRunner. When neither is set we + // omit --model and let pi pick its own startup default — we never + // inject a hardcoded id here. + var effectiveModel = !string.IsNullOrEmpty(modelId) ? modelId : DefaultModelId; + if (!string.IsNullOrEmpty(effectiveModel)) + { + argv.Add("--model"); + argv.Add(effectiveModel); + } + + // Reasoning effort maps 1:1 onto pi's --thinking flag. Only exact + // allowlist members are emitted; anything else (including pi's own + // future levels this list has not learned) is ignored rather than + // passed through to fail the CLI invocation. + if (!string.IsNullOrEmpty(reasoningMode) && ThinkingLevels.Contains(reasoningMode)) + { + argv.Add("--thinking"); + argv.Add(reasoningMode); + } + + // Pass the prompt via stdin rather than as a positional argv. + // Linux's MAX_ARG_STRLEN is 128 KiB per single argv element; rework + // prompts can exceed that. Verified against pi 0.85.1: `pi --mode + // json` with a piped-stdin prompt and no positional prompt arg emits + // the session header and proceeds to model resolution (it failed on + // auth, not on "no prompt"), so stdin alone is a complete prompt. + _ = captureStructuredStream; + _ = credential; + return new AgentInvocation(argv, Stdin: prompt); + } + + public override async Task RunAsync( + ISandbox sandbox, + string workingDirectory, + string prompt, + AgentCredential? credential, + string? modelId = null, + string? reasoningMode = null, + CancellationToken ct = default, + Action? stdoutChunkCallback = null, + bool captureStructuredStream = false) + { + var result = await base.RunAsync( + sandbox, + workingDirectory, + prompt, + credential, + modelId, + reasoningMode, + ct, + stdoutChunkCallback, + captureStructuredStream).ConfigureAwait(false); + + // Pi exits 0 on terminal run errors (verified: missing key, 401). + // Lift the terminal error region so the pipeline can classify it; + // without this an exit-0 quota/auth give-up with no file changes + // terminal-fails as "produced no changes". + if (string.IsNullOrEmpty(result.TerminalDiagnostic) + && PiTerminalDiagnoser.TryExtractTerminalError(result.Stdout) is { } terminalError) + { + return result with { TerminalDiagnostic = terminalError }; + } + + return result; + } + + protected override AgentInvocation BuildTextOnlyInvocation( + string prompt, + AgentCredential? credential, + string? modelId = null, + string? reasoningMode = null) + => BuildInvocation(prompt, credential, modelId, reasoningMode, captureStructuredStream: false); + + public string? GetTextOnlyUnavailabilityReason(AgentCredential? credential) + => GetSandboxSubscriptionTextOnlyUnavailabilityReason( + credential, + "ANTHROPIC_API_KEY"); + + // The pi CLI runs inside the work-item sandbox; a host-side text-only + // call with no sandbox returns failure (see RunTextOnlyRequiresSandboxAsync below). + public bool TextOnlyRequiresSandbox => true; + + public Task RunTextOnlyAsync( + string prompt, + AgentCredential? credential, + string? modelId = null, + string? reasoningMode = null, + CancellationToken ct = default, + ISandbox? sandbox = null, + string? workingDirectory = null) + { + if (sandbox is null || workingDirectory is null) + return RunTextOnlyRequiresSandboxAsync(ct); + + return ExecuteTextOnlyInSandboxAsync( + sandbox, + workingDirectory, + prompt, + credential, + modelId, + reasoningMode, + ct); + } +} diff --git a/src/CodeyBox.Agents.Pi/PiCostExtractor.cs b/src/CodeyBox.Agents.Pi/PiCostExtractor.cs new file mode 100644 index 00000000..d46cdfce --- /dev/null +++ b/src/CodeyBox.Agents.Pi/PiCostExtractor.cs @@ -0,0 +1,138 @@ +using System.Text.Json; +using CodeyBox.Agents; +using CodeyBox.Core; + +namespace CodeyBox.Agents.Pi; + +/// +/// Best-effort token-count extractor for pi's --mode json event stream. +/// +/// Every assistant message_start / message_end frame embeds +/// the provider-reported cumulative message.usage object +/// ({input, output, cacheRead, cacheWrite, totalTokens, cost:{…}} — +/// field names verified against pi 0.85.1's live error frames; the success +/// path carries the same object with non-zero counts). The extractor scans +/// every JSON line of stdout/stderr and keeps the LATEST usage object: usage +/// is cumulative per session, so the last frame is the run total. The dispatch +/// model id rides alongside on the same frames as message.model and is +/// recorded for per-model rate lookup. +/// +/// No is shipped: pi fronts 30+ providers +/// with unrelated per-token economics, so no single fallback rate is honest. +/// Per-model rates ship in agent-pricing-defaults.json or as operator +/// overrides under CodeyBox:AgentPricing, keyed pi/<model-id> +/// by the model id this extractor records. Unrated models cost $0 with a +/// startup warning (see AgentCostCalculator.ValidateAtStartup) — +/// mirroring the Cursor/Copilot subscription-path posture. +/// +public sealed class PiCostExtractor : IAgentCostExtractor +{ + public AgentKind Kind => AgentKind.Pi; + + public ModelRateConfig? DefaultPricing { get; } = null; + + // Cap on the model id recorded for cost attribution. Bounded so a + // pathological provider response can't blow past the schema column + // width on the downstream cost-summary table. + private const int MaxModelIdLength = 128; + + public AgentCostSnapshot? TryExtract(string? agentStdout, string? agentStderr) + { + try + { + var fromStdout = ScanStream(agentStdout); + var fromStderr = ScanStream(agentStderr); + // Prefer the stream with the larger reported total — stderr may + // carry a truncated retry while stdout holds the full session. + if (fromStdout is null) return fromStderr; + if (fromStderr is null) return fromStdout; + var stdoutTotal = fromStdout.InputTokens + fromStdout.CachedInputTokens + fromStdout.OutputTokens; + var stderrTotal = fromStderr.InputTokens + fromStderr.CachedInputTokens + fromStderr.OutputTokens; + return stderrTotal > stdoutTotal ? fromStderr : fromStdout; + } + catch (Exception) + { + // Contract: implementations must never throw. + return null; + } + } + + private readonly record struct ScannedUsage(int Input, int Cached, int Output, string? ModelId) + { + public int Total => Input + Cached + Output; + } + + private static AgentCostSnapshot? ScanStream(string? text) + { + if (string.IsNullOrWhiteSpace(text)) + return null; + + ScannedUsage? latest = null; + foreach (var rawLine in text.Split('\n')) + { + var line = rawLine.Trim(); + if (line.Length == 0 || !line.StartsWith('{')) + continue; + // Pre-screen: usage only rides on lines mentioning it, so the + // JSON parse below never runs on tool-call chatter. Uses + // Ordinal (not OrdinalIgnoreCase) — the wire field is lowercase + // `usage` and a case-insensitive match would also trip on prose + // like `"Usage: ..."` inside message text. + if (!line.Contains("\"usage\"", StringComparison.Ordinal)) + continue; + try + { + using var doc = JsonDocument.Parse(line); + var usage = ExtractUsage(doc.RootElement); + if (usage is not null) + latest = usage; + } + catch (JsonException) + { + // Interleaved non-JSON chatter — keep scanning. + } + } + + if (latest is not { } found) + return null; + if (found.Input == 0 && found.Cached == 0 && found.Output == 0) + return null; + return new AgentCostSnapshot(found.Input, found.Cached, found.Output, found.ModelId); + } + + private static ScannedUsage? ExtractUsage(JsonElement root) + { + if (root.ValueKind != JsonValueKind.Object) + return null; + + // Usage rides on the nested assistant message; the top-level + // turn/agent frames repeat it via "message". + var message = root; + if (root.TryGetProperty("message", out var nested) && nested.ValueKind == JsonValueKind.Object) + message = nested; + + if (!message.TryGetProperty("usage", out var usage) || usage.ValueKind != JsonValueKind.Object) + return null; + + var input = ReadNonNegative(usage, "input"); + var cached = ReadNonNegative(usage, "cacheRead"); + var output = ReadNonNegative(usage, "output"); + + string? modelId = null; + if (message.TryGetProperty("model", out var model) && model.ValueKind == JsonValueKind.String) + { + var raw = model.GetString(); + if (!string.IsNullOrWhiteSpace(raw)) + modelId = raw!.Length > MaxModelIdLength ? raw[..MaxModelIdLength] : raw; + } + + return new ScannedUsage(input, cached, output, modelId); + } + + private static int ReadNonNegative(JsonElement obj, string name) + { + if (obj.TryGetProperty(name, out var value) && value.TryGetInt32(out var n) && n > 0) + return n; + return 0; + } +} diff --git a/src/CodeyBox.Agents.Pi/PiInVmSmokeProbe.cs b/src/CodeyBox.Agents.Pi/PiInVmSmokeProbe.cs new file mode 100644 index 00000000..e8e3d147 --- /dev/null +++ b/src/CodeyBox.Agents.Pi/PiInVmSmokeProbe.cs @@ -0,0 +1,41 @@ +using CodeyBox.Core; + +namespace CodeyBox.Agents.Pi; + +/// +/// In-VM smoke check for the pi CLI: +/// +/// pi --version — binary present on PATH (exit 127 otherwise). +/// pi --help must advertise --mode/json — the +/// runner's only transport. A pi build that dropped the JSON event stream +/// would otherwise dispatch into an unparseable run and fail late. +/// +/// +/// No auth step: pi supports 30+ providers with no single lightweight +/// "whoami", and any provider call would spend real quota. Credential +/// viability is covered host-side by ; the first +/// real dispatch surfaces provider auth errors through +/// . +/// +public sealed class PiInVmSmokeProbe : IInVmSmokeProbe +{ + public AgentKind Kind => AgentKind.Pi; + + public IReadOnlyList BuildSteps(AgentCredential? credential) + { + _ = credential; + return + [ + new( + [PiAgentRunner.DefaultBinary, "--version"], + FailureHint: "pi binary not runnable on sandbox PATH"), + // Exit-code-only by InVmSmokeStep contract, so assert --mode/json + // support through grep's exit code: the runner's only transport is + // the JSON event stream, and a pi build that dropped it must bench + // here rather than dispatch into an unparseable run. + new( + ["bash", "-c", $"{PiAgentRunner.DefaultBinary} --help | grep -q -- --mode"], + FailureHint: "pi --help does not advertise --mode; --mode json support could not be verified"), + ]; + } +} diff --git a/src/CodeyBox.Agents.Pi/PiKnownModels.cs b/src/CodeyBox.Agents.Pi/PiKnownModels.cs new file mode 100644 index 00000000..2b2c137f --- /dev/null +++ b/src/CodeyBox.Agents.Pi/PiKnownModels.cs @@ -0,0 +1,71 @@ +using Microsoft.Extensions.Logging; + +namespace CodeyBox.Agents.Pi; + +/// +/// Seed list of pi model ids driving the warn-only config validator and the +/// . +/// +/// Pi has no host-queryable model catalog: pi --list-models needs +/// an authenticated provider and network, and the catalog refreshes +/// automatically per provider — so the probe cannot live-read it the way the +/// opencode probe shells out to opencode models. This list is therefore +/// a curated seed, not an enumeration: every entry has a provenance comment +/// (live dispatch verification or pi's own --help/README examples), and +/// unknown ids are never rejected — the CLI accepts fuzzy patterns and +/// provider/id-qualified ids far beyond this seed, so validation only +/// warns (mirroring AntigravityKnownModels). +/// +public static class PiKnownModels +{ + /// + /// Curated seed of pi-accepted model ids. Prefer + /// provider/id-qualified ids in operator config: pi's default + /// provider is google, so a bare id can resolve against the wrong + /// provider catalog. + /// + public static readonly IReadOnlyList All = new[] + { + // Live-verified 2026-09-14 against pi 0.85.1: `--provider anthropic + // --model claude-haiku-4-5` with a bogus key reached api.anthropic.com + // and returned a provider 401, proving the model id passed CLI-side + // resolution. + "anthropic/claude-haiku-4-5", + // pi --help model examples (provider-qualified form). + "openai/gpt-4o", + "openai/gpt-4o-mini", + }; + + public static bool IsKnown(string? modelId) + { + if (string.IsNullOrWhiteSpace(modelId)) return false; + foreach (var m in All) + { + if (string.Equals(m, modelId, StringComparison.OrdinalIgnoreCase)) + return true; + } + return false; + } + + /// + /// Logs a warning when the operator-configured + /// for a pi member is not in . Unknown ids are not + /// rejected — pi accepts fuzzy patterns and provider-qualified ids beyond + /// this seed — but the warning prompts operators to double-check typos + /// before a dispatch fails at model-resolution time. + /// + public static string? ValidateModelIdAgainstProviderList( + string classId, string? modelId, ILogger log) + { + if (string.IsNullOrWhiteSpace(modelId)) return null; + if (IsKnown(modelId)) return null; + var message = $"AgentClass '{classId}': Pi member ModelId '{modelId}' is not in the known " + + $"provider list ({string.Join(", ", All)}). Pi accepts fuzzy and provider-qualified ids beyond this seed; " + + "double-check for a typo."; + log.LogWarning( + "AgentClass '{ClassId}': Pi member ModelId '{ModelId}' is not in the known provider list ({Known}). " + + "Pi accepts fuzzy and provider-qualified ids beyond this seed; double-check for a typo.", + classId, modelId, string.Join(", ", All)); + return message; + } +} diff --git a/src/CodeyBox.Agents.Pi/PiModelListProbe.cs b/src/CodeyBox.Agents.Pi/PiModelListProbe.cs new file mode 100644 index 00000000..e10bffbf --- /dev/null +++ b/src/CodeyBox.Agents.Pi/PiModelListProbe.cs @@ -0,0 +1,20 @@ +using CodeyBox.Core; + +namespace CodeyBox.Agents.Pi; + +/// +/// Model-list probe for pi. Returns the curated +/// seed: pi's live catalog (pi --list-models) requires an authenticated +/// provider plus network, so it cannot back a host-side startup probe the way +/// the opencode probe shells out to opencode models. Operator-configured +/// ids absent from the seed surface as a startup warning, not a hard reject +/// (see ). +/// Mirrors the Crock/Antigravity static-list probes. +/// +public sealed class PiModelListProbe : IAgentModelListProbe +{ + public AgentKind Kind => AgentKind.Pi; + + public Task GetModelListAsync(CancellationToken ct) + => Task.FromResult(AgentModelListResult.Success(PiKnownModels.All)); +} diff --git a/src/CodeyBox.Agents.Pi/PiQuotaFailureDetector.cs b/src/CodeyBox.Agents.Pi/PiQuotaFailureDetector.cs new file mode 100644 index 00000000..ae82cc03 --- /dev/null +++ b/src/CodeyBox.Agents.Pi/PiQuotaFailureDetector.cs @@ -0,0 +1,120 @@ +using CodeyBox.Agents; +using CodeyBox.Core; + +namespace CodeyBox.Agents.Pi; + +/// +/// Recognises quota / rate-limit / auth failures emitted by the pi CLI. +/// +/// Pi relays the backing provider's error verbatim inside the JSON +/// event's errorMessage (verified against pi 0.85.1: a bad Anthropic key +/// yields 401 {"type":"error","error":{"type":"authentication_error",…}}), +/// and emits a plaintext No API key found for the selected model. when +/// it cannot start at all. Because --mode json stdout carries these +/// shapes, the detector scans BOTH streams — the terminal frames live on +/// stdout, not stderr. +/// +/// The pattern list is config-driven: built-in defaults ship in +/// , and operators can append additional patterns +/// via CodeyBox:QuotaFailurePatterns:pi without recompilation (mirroring +/// the cursor detector). Patterns stay anchored to provider-shaped phrases +/// (HTTP status text, *_error codes, full sentences) rather than bare +/// numbers or single words: pi prompts can contain repository content under +/// review, and model output citing "429" or discussing quota code must not +/// gate dispatch. +/// +public sealed class PiQuotaFailureDetector : IAgentQuotaFailureDetector +{ + public AgentKind Kind => AgentKind.Pi; + + /// + /// Built-in pattern allowlist. Order matters: rate-limit rows come first + /// so a refusal carrying both shapes parks on the rate-limit backoff + /// rather than terminating the item as a hard quota failure. + /// + public static readonly IReadOnlyList DefaultPatterns = + [ + // Shared provider rate-limit rows (transient throughput refusals). + .. SharedRateLimitPatterns.ProviderRateLimitPatterns, + // Pre-session plaintext failure (exit 0, no JSON error event). + new("No API key found", QuotaFailureKind.Unauthorized), + // Provider-shaped auth failures relayed verbatim in errorMessage. + // "authentication_error" is Anthropic's code; the "API key is invalid" + // sentence is its human message. Both are provider vocabulary, not + // words model output plausibly emits about code under review. + new("authentication_error", QuotaFailureKind.Unauthorized), + new("API key is invalid", QuotaFailureKind.Unauthorized), + new("invalid_api_key", QuotaFailureKind.Unauthorized), + new("incorrect api key", QuotaFailureKind.Unauthorized), + new("401 Unauthorized", QuotaFailureKind.Unauthorized), + new("API Error: 401", QuotaFailureKind.Unauthorized), + // Billing / hard-cap exhaustion relayed verbatim. + new("insufficient_quota", QuotaFailureKind.LimitReached), + new("insufficient credits", QuotaFailureKind.LimitReached), + new("billing_hard_limit_reached", QuotaFailureKind.LimitReached), + new("HTTP 402", QuotaFailureKind.LimitReached), + new("402 Payment Required", QuotaFailureKind.LimitReached), + // "quota" alone matches reviewing-quota-code text; require a verb + // that conveys exhaustion. + new("quota exceeded", QuotaFailureKind.LimitReached), + new("quota exhausted", QuotaFailureKind.LimitReached), + new("usage limit reached", QuotaFailureKind.LimitReached), + ]; + + private readonly IReadOnlyList _patterns; + + /// + /// Constructs a detector with the built-in . + /// + public PiQuotaFailureDetector() : this(additionalPatterns: null) { } + + /// + /// Constructs a detector whose pattern list is + /// followed by . Operator-configured + /// patterns are checked after defaults; null/empty input behaves + /// identically to the parameterless constructor. + /// + public PiQuotaFailureDetector(IEnumerable? additionalPatterns) + { + if (additionalPatterns is null) + { + _patterns = DefaultPatterns; + return; + } + + var extras = additionalPatterns.Where(p => !string.IsNullOrEmpty(p.Pattern)).ToArray(); + if (extras.Length == 0) + { + _patterns = DefaultPatterns; + return; + } + + var combined = new List(DefaultPatterns.Count + extras.Length); + combined.AddRange(DefaultPatterns); + combined.AddRange(extras); + _patterns = combined; + } + + public QuotaDetection? Detect(string? stderr, string? stdout) + { + if (string.IsNullOrEmpty(stderr) && string.IsNullOrEmpty(stdout)) + return null; + + foreach (var entry in _patterns) + { + var inStderr = !string.IsNullOrEmpty(stderr) && stderr.Contains(entry.Pattern, StringComparison.OrdinalIgnoreCase); + var inStdout = !string.IsNullOrEmpty(stdout) && stdout.Contains(entry.Pattern, StringComparison.OrdinalIgnoreCase); + if (!inStderr && !inStdout) continue; + + var resetSources = new List(2); + if (!string.IsNullOrEmpty(stderr)) resetSources.Add(stderr); + if (!string.IsNullOrEmpty(stdout)) resetSources.Add(stdout); + return new QuotaDetection( + entry.Kind, + QuotaResetParser.TryParseResetAt(resetSources) + ?? QuotaResetParser.TryParseRetryAfterHeader(resetSources)); + } + + return null; + } +} diff --git a/src/CodeyBox.Agents.Pi/PiSmokeProbe.cs b/src/CodeyBox.Agents.Pi/PiSmokeProbe.cs new file mode 100644 index 00000000..a04e318d --- /dev/null +++ b/src/CodeyBox.Agents.Pi/PiSmokeProbe.cs @@ -0,0 +1,49 @@ +using CodeyBox.Core; +using Microsoft.Extensions.Logging; + +namespace CodeyBox.Agents.Pi; + +/// +/// Minimal credential viability check for pi. Returns Ok when the bundle +/// contains a non-empty ANTHROPIC_API_KEY (the shipped credential +/// mapping wires host CODEYBOX_PI_API_KEY to it); returns Fail +/// otherwise. +/// +/// Unlike the Claude / Codex / Gemini probes this does NOT issue a +/// network call: pi fronts 30+ providers behind one CLI, so no single +/// endpoint validates "the" credential, and probing an arbitrary provider +/// endpoint would burn quota on a meter the operator may not even use. The +/// real auth check happens on first CLI call inside the sandbox, where the +/// runner lifts the terminal error (see ). +/// Mirrors the Cursor/Opencode presence-check probes. +/// +public sealed class PiSmokeProbe : IAgentSmokeProbe +{ + private readonly ILogger? _log; + + public AgentKind Kind => AgentKind.Pi; + + public PiSmokeProbe(ILogger? log = null) + { + _log = log; + } + + public Task SmokeTestAsync(AgentCredential credential, CancellationToken ct) + { + // No I/O happens here — report TimeSpan.Zero explicitly so the value + // is honest about what was measured, matching the OpencodeSmokeProbe + // convention a future network-backed probe can build on. + var hasApiKey = credential.EnvironmentVariables.TryGetValue("ANTHROPIC_API_KEY", out var key) + && !string.IsNullOrEmpty(key); + if (!hasApiKey) + { + _log?.LogDebug("Pi smoke probe found no ANTHROPIC_API_KEY in credential bundle"); + return Task.FromResult(new AgentSmokeResult( + false, + "no Pi credential configured (set host CODEYBOX_PI_API_KEY)", + TimeSpan.Zero, + SmokeFailureCategory.Persistent)); + } + return Task.FromResult(new AgentSmokeResult(true, null, TimeSpan.Zero, SmokeFailureCategory.None)); + } +} diff --git a/src/CodeyBox.Agents.Pi/PiStreamParser.cs b/src/CodeyBox.Agents.Pi/PiStreamParser.cs new file mode 100644 index 00000000..2f047c63 --- /dev/null +++ b/src/CodeyBox.Agents.Pi/PiStreamParser.cs @@ -0,0 +1,113 @@ +using System.Text.Json; +using CodeyBox.Agents; +using CodeyBox.Core; + +namespace CodeyBox.Agents.Pi; + +/// +/// Stream parser for pi's --mode json event lines. +/// +/// Claim vocabulary (verified against pi 0.85.1 live frames). The +/// session header {"type":"session","version":3,"id":…,"cwd":…} and the +/// underscore-style lifecycle verbs (agent_start, turn_start, +/// message_start, message_update, message_end, +/// turn_end, agent_end, agent_settled) are pi-specific: +/// no other registered parser claims them (Claude claims +/// assistant/user/result/tool_use/tool_result; Codex claims dotted +/// thread./turn./item. prefixes; Gemini keys on +/// usageMetadata). The claim additionally requires the header's numeric +/// version + string cwd on type: "session" so a foreign +/// line that merely reuses the word "session" is never stolen. +/// +/// Usage mapping. The shared ParseUsage only knows +/// input_tokens/prompt_tokens-style names; pi reports +/// usage:{input, output, cacheRead, cacheWrite, totalTokens}, so this +/// parser supplements the base parse with pi's names (base results win when +/// both are present). cacheWrite has no cost-bucket in +/// and is ignored for token accounting — +/// same treatment as the other providers. +/// +public sealed class PiStreamParser : FlexibleAgentStreamParser +{ + public PiStreamParser(AgentStreamParserOptions? options = null) + : base(AgentKind.Pi, options) + { + } + + private static readonly HashSet PiLifecycleTypes = new(StringComparer.Ordinal) + { + "session", + "agent_start", + "agent_end", + "agent_settled", + "turn_start", + "turn_end", + "message_start", + "message_update", + "message_end", + "queue_update", + "compaction_start", + "compaction_end", + }; + + public override bool TryClaim(JsonElement line) + => IsPiStreamJsonEvent(line); + + internal static bool IsPiStreamJsonEvent(JsonElement line) + { + if (line.ValueKind != JsonValueKind.Object) + return false; + if (!line.TryGetProperty("type", out var typeProp) + || typeProp.ValueKind != JsonValueKind.String) + return false; + + var type = typeProp.GetString(); + if (string.IsNullOrEmpty(type) || !PiLifecycleTypes.Contains(type)) + return false; + + // The bare lifecycle verbs are already pi-unique, but the session + // header gets a shape check too: require the numeric version and cwd + // the live header carries, so a foreign {"type":"session"} line is + // never misattributed. + if (string.Equals(type, "session", StringComparison.Ordinal)) + { + var hasVersion = line.TryGetProperty("version", out var version) + && version.ValueKind == JsonValueKind.Number; + var hasCwd = line.TryGetProperty("cwd", out var cwd) + && cwd.ValueKind == JsonValueKind.String; + return hasVersion && hasCwd; + } + + return true; + } + + protected override ParsedEvent ParseEvent(JsonElement root) + { + var parsed = base.ParseEvent(root); + + // Supplement (never override) the base usage parse with pi's field + // names. The usage object rides on the nested assistant message; + // top-level turn/agent frames repeat it via "message". + var message = root; + if (TryGet(root, out var nested, "message") && nested.ValueKind == JsonValueKind.Object) + message = nested; + + if (TryGet(message, out var usage, "usage") && usage.ValueKind == JsonValueKind.Object) + { + var input = parsed.InputTokens ?? FirstNullableInt(usage, "input"); + var output = parsed.OutputTokens ?? FirstNullableInt(usage, "output"); + var cached = parsed.CachedInputTokens ?? FirstNullableInt(usage, "cacheRead"); + if (input != parsed.InputTokens || output != parsed.OutputTokens || cached != parsed.CachedInputTokens) + parsed = parsed with { InputTokens = input, OutputTokens = output, CachedInputTokens = cached }; + } + + return parsed; + } + + private static int? FirstNullableInt(JsonElement obj, string name) + { + if (obj.TryGetProperty(name, out var value) && value.TryGetInt32(out var n) && n >= 0) + return n; + return null; + } +} diff --git a/src/CodeyBox.Agents.Pi/PiTerminalDiagnoser.cs b/src/CodeyBox.Agents.Pi/PiTerminalDiagnoser.cs new file mode 100644 index 00000000..11940888 --- /dev/null +++ b/src/CodeyBox.Agents.Pi/PiTerminalDiagnoser.cs @@ -0,0 +1,102 @@ +using System.Text.Json; + +namespace CodeyBox.Agents.Pi; + +/// +/// Pure extraction of pi's terminal run error from --mode json stdout. +/// +/// Pi exits 0 even when the run dies before producing output (verified +/// against pi 0.85.1: a missing API key and a provider 401 both exit 0 with +/// the cause only in the event stream). The failure surfaces two ways: +/// +/// A structured assistant message with +/// stopReason: "error" and errorMessage (e.g. +/// 401 {"type":"error","error":{"type":"authentication_error",…}}), +/// repeated across message_end / turn_end / agent_end — +/// the first one wins. +/// A plaintext pre-session line, +/// No API key found for the selected model., emitted when pi cannot +/// even start the run (no JSON error event follows). +/// +/// +/// Returns null when no terminal error is present (a healthy run's +/// agent_end carries "willRetry":false with no +/// stopReason). Never throws: malformed lines are skipped so a +/// half-written stream still yields whatever terminal signal it contains. +/// Output is capped at so the pipeline's +/// audit/webhook sinks never balloon on a verbose provider error body. +/// +internal static class PiTerminalDiagnoser +{ + internal const int MaxDiagnosticChars = 500; + + internal static string? TryExtractTerminalError(string? stdout) + { + if (string.IsNullOrWhiteSpace(stdout)) + return null; + + foreach (var rawLine in stdout.Split('\n')) + { + var line = rawLine.Trim(); + if (line.Length == 0) + continue; + + // Pre-session plaintext failure carries no JSON framing. + if (line.Contains("No API key found", StringComparison.OrdinalIgnoreCase)) + return Truncate(line); + + if (!line.StartsWith('{')) + continue; + + try + { + using var doc = JsonDocument.Parse(line); + if (TryExtractStructuredError(doc.RootElement) is { } error) + return error; + } + catch (JsonException) + { + // Half-written or interleaved chatter — keep scanning. + } + } + + return null; + } + + private static string? TryExtractStructuredError(JsonElement root) + { + if (root.ValueKind != JsonValueKind.Object) + return null; + + // The error rides on the nested assistant message (message_end / + // turn_end / agent_end all repeat it); the top-level turn/agent frames + // have no stopReason of their own. + var message = root; + if (root.TryGetProperty("message", out var nested) && nested.ValueKind == JsonValueKind.Object) + message = nested; + + if (!message.TryGetProperty("stopReason", out var stopReason) + || stopReason.ValueKind != JsonValueKind.String + || !string.Equals(stopReason.GetString(), "error", StringComparison.OrdinalIgnoreCase)) + { + return null; + } + + if (message.TryGetProperty("errorMessage", out var errorMessage) + && errorMessage.ValueKind == JsonValueKind.String + && !string.IsNullOrWhiteSpace(errorMessage.GetString())) + { + return Truncate(errorMessage.GetString()!); + } + + return Truncate("pi run ended with stopReason=error (no errorMessage)"); + } + + private static string Truncate(string value) + { + var trimmed = value.Trim(); + return trimmed.Length <= MaxDiagnosticChars + ? trimmed + : trimmed[..MaxDiagnosticChars] + "…"; + } +} diff --git a/src/CodeyBox.Api/AgentClassesConfigBuilder.cs b/src/CodeyBox.Api/AgentClassesConfigBuilder.cs index 0d1c48ea..e309d97b 100644 --- a/src/CodeyBox.Api/AgentClassesConfigBuilder.cs +++ b/src/CodeyBox.Api/AgentClassesConfigBuilder.cs @@ -1,6 +1,7 @@ using CodeyBox.Agents.Antigravity; using CodeyBox.Agents.Copilot; using CodeyBox.Agents.Gemini; +using CodeyBox.Agents.Pi; using CodeyBox.Core; using CodeyBox.Orchestrator; using Microsoft.Extensions.Logging; @@ -90,6 +91,8 @@ public static IReadOnlyList Build( GeminiKnownModels.ValidateModelIdAgainstProviderList(classOpts.Id, agentKind, m.ModelId, log); if (agentKind == AgentKind.Antigravity) AntigravityKnownModels.ValidateModelIdAgainstProviderList(classOpts.Id, m.ModelId, log); + if (agentKind == AgentKind.Pi) + PiKnownModels.ValidateModelIdAgainstProviderList(classOpts.Id, m.ModelId, log); // Capabilities are operator-declared tags. Normalise (trim + drop empties) // and de-duplicate case-insensitively so '"sensitive"' and '"Sensitive"' // don't both end up in the list. Tag values themselves are otherwise diff --git a/src/CodeyBox.Api/CodeyBox.Api.csproj b/src/CodeyBox.Api/CodeyBox.Api.csproj index 8e919381..92610bb8 100644 --- a/src/CodeyBox.Api/CodeyBox.Api.csproj +++ b/src/CodeyBox.Api/CodeyBox.Api.csproj @@ -15,6 +15,7 @@ + diff --git a/src/CodeyBox.Api/Program.cs b/src/CodeyBox.Api/Program.cs index eafc9f88..a9b32d31 100644 --- a/src/CodeyBox.Api/Program.cs +++ b/src/CodeyBox.Api/Program.cs @@ -14,6 +14,7 @@ using CodeyBox.Agents.Cursor; using CodeyBox.Agents.Gemini; using CodeyBox.Agents.Opencode; +using CodeyBox.Agents.Pi; using CodeyBox.AdminSeed; using CodeyBox.Api; using CodeyBox.Api.Hubs; @@ -1172,6 +1173,15 @@ static string FormatBytes(long bytes) { SandboxOptions = () => sp.GetRequiredService>().CurrentValue.Crock, }); +// Pi: lean terminal coding agent (npm @earendil-works/pi-coding-agent, MIT). +// Driven via `pi --mode json` (structured event stream, not raw -p and not +// --mode rpc) with the prompt on stdin. Auth is a provider API key from the +// environment (shipped mapping: CODEYBOX_PI_API_KEY -> ANTHROPIC_API_KEY). +// The binary must be installed in the sandbox image +// (`npm install -g --ignore-scripts @earendil-works/pi-coding-agent`); see +// docs/concepts/agents.md and docs/reference/agent-quirks.md. +builder.Services.AddSingleton(sp => new PiAgentRunner( + sp.GetRequiredService())); // Seeded fake-agent run mode for the admin E2E/demo instance (see // docs/concepts/admin-e2e.md). Opt-in via CodeyBox:SeededFakeAgents:Enabled; // when disabled nothing here registers and production routing is untouched. @@ -1589,6 +1599,12 @@ static string FormatBytes(long bytes) // Note: Antigravity is NOT in this verbatim mapping. The agy CLI's // OAuth token bundle is shipped to the sandbox by the dedicated // AntigravityEnvironmentCredentialProvider registered separately below. + // Pi: provider API-key auth from the environment. Pi reads the + // provider-native variable (ANTHROPIC_API_KEY, OPENAI_API_KEY, + // GEMINI_API_KEY, … — full table in pi's providers.md); the shipped + // mapping covers the Anthropic path. Operators fronting other + // providers add that provider's variable here following the same row. + new AgentCredentialMapping(AgentKind.Pi, "CODEYBOX_PI_API_KEY", "ANTHROPIC_API_KEY"), })); // Antigravity uses Sign-in-with-Google OAuth. The dedicated provider ships // the agy token bundle verbatim (refresh_token RETAINED) into the sandbox, @@ -2273,6 +2289,13 @@ static IAgentQuotaProbe WireQuotaProbeTokenInvalidation( new AntigravitySmokeProbe( sp.GetRequiredService(), sp.GetRequiredService().CreateLogger())); +// Pi: credential-presence check only (ANTHROPIC_API_KEY in the bundle). Pi +// fronts 30+ providers behind one CLI, so no single endpoint validates the +// credential and any provider call would spend real quota; the real auth +// check happens on first CLI call in-VM. +builder.Services.AddSingleton(sp => + new PiSmokeProbe( + sp.GetRequiredService().CreateLogger())); // --- In-VM smoke probes ------------------------------------------------------ // Registered as IEnumerable; InVmSmokeProber resolves by Kind. @@ -2286,6 +2309,7 @@ static IAgentQuotaProbe WireQuotaProbeTokenInvalidation( builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); +builder.Services.AddSingleton(); // Startup guard (AC#1): bench any configured AgentClass member with no in-VM // probe (so a CLI-backed agent that would fail at first dispatch is routed past // at smoke time, not first dispatch). Agents on @@ -2363,6 +2387,11 @@ static IAgentQuotaProbe WireQuotaProbeTokenInvalidation( // CrockKnownModels list. Operator-configured ids that are absent surface as // a startup warning, not a hard reject. builder.Services.AddSingleton(); +// Pi model-list probe: `pi --list-models` needs an authenticated provider +// plus network, so it cannot back a host-side startup probe. The curated +// PiKnownModels seed is authoritative; operator-configured ids absent from +// the seed surface as a startup warning, not a hard reject. +builder.Services.AddSingleton(); builder.Services.AddHostedService(); builder.Services.AddSingleton(sp => @@ -3346,6 +3375,7 @@ static Func DotnetTestRunOptionsAccessor(IServiceProvider sp) [AgentKind.Copilot] = new CopilotCostExtractor(), [AgentKind.Antigravity] = new AntigravityCostExtractor(), [AgentKind.Crock] = new CrockCostExtractor(), + [AgentKind.Pi] = new PiCostExtractor(), }; // Warn once at startup for registered agents with no extractor. foreach (var kind in registry.Available) @@ -3437,6 +3467,7 @@ static Func DotnetTestRunOptionsAccessor(IServiceProvider sp) builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); +builder.Services.AddSingleton(); builder.Services.AddSingleton(); // Per-provider buffered-stdout tool-call counters. Used by the orchestrator @@ -3472,6 +3503,21 @@ static Func DotnetTestRunOptionsAccessor(IServiceProvider sp) return new CursorQuotaFailureDetector(extras); }); builder.Services.AddSingleton(); +builder.Services.AddSingleton(sp => +{ + // Pi detector accepts operator-extensible patterns from + // CodeyBox:QuotaFailurePatterns:pi, mirroring the cursor hook above. + var cbOpts = sp.GetRequiredService>().Value; + var extras = cbOpts.QuotaFailurePatterns is null + ? null + : cbOpts.QuotaFailurePatterns + .Where(kvp => string.Equals(kvp.Key, AgentKind.Pi.Value, StringComparison.OrdinalIgnoreCase)) + .SelectMany(kvp => kvp.Value ?? new List()) + .Where(p => !string.IsNullOrEmpty(p.Pattern)) + .Select(p => new QuotaFailurePattern(p.Pattern, p.Kind)) + .ToArray(); + return new PiQuotaFailureDetector(extras); +}); builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(sp => diff --git a/src/CodeyBox.Api/agent-pricing-defaults.json b/src/CodeyBox.Api/agent-pricing-defaults.json index 8cf7d40e..f7d2e6c8 100644 --- a/src/CodeyBox.Api/agent-pricing-defaults.json +++ b/src/CodeyBox.Api/agent-pricing-defaults.json @@ -5,7 +5,8 @@ "opencode": "https://opencode.ai/docs/go", "claude": "https://docs.anthropic.com/en/docs/about-claude/pricing", "codex": "https://openai.com/api/pricing/", - "gemini": "https://ai.google.dev/gemini-api/docs/pricing" + "gemini": "https://ai.google.dev/gemini-api/docs/pricing", + "pi": "https://docs.anthropic.com/en/docs/about-claude/pricing" }, "notes": { "opencode": "OpenCode Go is subscription-priced ($12 per 5h usage budget on https://opencode.ai/docs/go). Each model entry uses one subscription-equivalent USD/M rate for input, cached input, and output: ($12 ÷ requests-per-5h) ÷ ((typical input + cached + output tokens per request) ÷ 1e6), using the request limits and typical token mix published on that page. Keys use opencode-go/. Cost attribution charges the stored non-cached input bucket and cached input bucket separately; public total-input reporting adds them.", @@ -14,7 +15,8 @@ "gemini": "gemini-3-flash-preview is the default AgentClasses model. Pro preview models use short-context rates (<= 200k tokens); long-context tier (>200k) is not represented.", "cursor": "Cursor uses a flat subscription and does not publish per-token rates; intentionally omitted from defaults.", "copilot": "Copilot uses a flat subscription and does not publish per-token rates; intentionally omitted from defaults.", - "crock": "CrockCode is a pay-per-token Anthropic API-key path that rides the Message Batches API (~50% discount applied by Anthropic at billing time). Rates below are the post-batch-discount effective rates (half of the on-demand /v1/messages rate for the same model id). Keys mirror the canonical Anthropic model ids. Cost attribution is an ESTIMATE, not exact spend: like the claude bucket, cache-write (cache_creation) tokens are folded into fresh input at the base rate and Anthropic's 1.25x/2x cache-write premium is not represented. DefaultRates.crock is the Opus-tier upper-bound fallback for any model id not enumerated here." + "crock": "CrockCode is a pay-per-token Anthropic API-key path that rides the Message Batches API (~50% discount applied by Anthropic at billing time). Rates below are the post-batch-discount effective rates (half of the on-demand /v1/messages rate for the same model id). Keys mirror the canonical Anthropic model ids. Cost attribution is an ESTIMATE, not exact spend: like the claude bucket, cache-write (cache_creation) tokens are folded into fresh input at the base rate and Anthropic's 1.25x/2x cache-write premium is not represented. DefaultRates.crock is the Opus-tier upper-bound fallback for any model id not enumerated here.", + "pi": "Pi is a multi-provider front; rates depend on the backing provider. The shipped AgentClasses member routes Anthropic, so this bucket mirrors the Anthropic on-demand rates for the models operators route through pi. Keys are the bare provider model ids pi reports in message.model (pi strips the provider/ qualifier), so operator overrides use the same bare form. Operators fronting other providers add that provider's list prices here (or under CodeyBox:AgentPricing) keyed by the bare message.model id." } }, "DefaultRates": { @@ -61,6 +63,9 @@ "claude-opus-4-6": { "inputPerMillion": 2.50, "cachedInputPerMillion": 0.25, "outputPerMillion": 12.50 }, "claude-sonnet-4-6": { "inputPerMillion": 1.50, "cachedInputPerMillion": 0.15, "outputPerMillion": 7.50 }, "claude-haiku-4-5": { "inputPerMillion": 0.50, "cachedInputPerMillion": 0.05, "outputPerMillion": 2.50 } + }, + "pi": { + "claude-haiku-4-5": { "inputPerMillion": 1.00, "cachedInputPerMillion": 0.10, "outputPerMillion": 5.00 } } } } diff --git a/src/CodeyBox.Api/appsettings.json b/src/CodeyBox.Api/appsettings.json index bb9f0b02..64b4b74c 100644 --- a/src/CodeyBox.Api/appsettings.json +++ b/src/CodeyBox.Api/appsettings.json @@ -137,7 +137,8 @@ { "Agent": "claude", "Billing": "Subscription", "ModelId": "claude-opus-4-7", "QualityScore": 100, "Capabilities": [ "sensitive" ] }, { "Agent": "codex", "Billing": "Subscription", "ModelId": "gpt-5.5", "QualityScore": 100, "Capabilities": [ "sensitive" ] }, { "Agent": "cursor", "Billing": "Subscription", "ModelId": "composer-2.5", "QualityScore": 98 }, - { "Agent": "gemini", "Billing": "Subscription", "ModelId": "gemini-3-flash-preview", "QualityScore": 95, "ReasoningMode": "high" } + { "Agent": "gemini", "Billing": "Subscription", "ModelId": "gemini-3-flash-preview", "QualityScore": 95, "ReasoningMode": "high" }, + { "Agent": "pi", "Billing": "PayPerApi", "ModelId": "anthropic/claude-haiku-4-5", "QualityScore": 90 } ] } ], diff --git a/src/CodeyBox.Core/AgentKind.cs b/src/CodeyBox.Core/AgentKind.cs index c75241b2..d9bdafc0 100644 --- a/src/CodeyBox.Core/AgentKind.cs +++ b/src/CodeyBox.Core/AgentKind.cs @@ -14,6 +14,7 @@ public readonly record struct AgentKind(string Value) public static AgentKind Opencode { get; } = new("opencode"); public static AgentKind Antigravity { get; } = new("antigravity"); public static AgentKind Crock { get; } = new("crock"); + public static AgentKind Pi { get; } = new("pi"); public override string ToString() => Value; } diff --git a/tests/CodeyBox.Tests/CodeyBox.Tests.csproj b/tests/CodeyBox.Tests/CodeyBox.Tests.csproj index f649fd19..8c43b054 100644 --- a/tests/CodeyBox.Tests/CodeyBox.Tests.csproj +++ b/tests/CodeyBox.Tests/CodeyBox.Tests.csproj @@ -69,6 +69,7 @@ + diff --git a/tests/CodeyBox.Tests/InVmSmokeProbeBuildStepsTests.cs b/tests/CodeyBox.Tests/InVmSmokeProbeBuildStepsTests.cs index dc13e31a..a7c62b50 100644 --- a/tests/CodeyBox.Tests/InVmSmokeProbeBuildStepsTests.cs +++ b/tests/CodeyBox.Tests/InVmSmokeProbeBuildStepsTests.cs @@ -5,6 +5,7 @@ using CodeyBox.Agents.Cursor; using CodeyBox.Agents.Gemini; using CodeyBox.Agents.Opencode; +using CodeyBox.Agents.Pi; using CodeyBox.Core; namespace CodeyBox.Tests; @@ -111,4 +112,23 @@ public void Opencode_WithAuthEnv_EmitsVersionMaterialiseProviders() Assert.Equal(["bash", "-c", OpencodeAgentRunner.AuthMaterialiseScript], steps[1].Argv); Assert.Equal([OpencodeAgentRunner.DefaultBinary, "providers"], steps[2].Argv); } + + [Fact] + public void Pi_EmitsVersionPlusModeJsonAssertion_PinnedToRunnerBinary() + { + // Pi's probe has two steps: the --version binary check plus a + // --mode assertion (the runner's only transport). Both pin to the + // runner's binary constant so probe/runner drift fails loudly. + var probe = new PiInVmSmokeProbe(); + Assert.Equal(AgentKind.Pi, probe.Kind); + + foreach (var credential in new AgentCredential?[] { null, Cred(AgentKind.Pi) }) + { + var steps = probe.BuildSteps(credential); + Assert.Equal(2, steps.Count); + Assert.Equal([PiAgentRunner.DefaultBinary, "--version"], steps[0].Argv); + Assert.Contains(PiAgentRunner.DefaultBinary, string.Join(" ", steps[1].Argv)); + Assert.Contains("--mode", string.Join(" ", steps[1].Argv)); + } + } } diff --git a/tests/CodeyBox.Tests/PiAgentRunnerTests.cs b/tests/CodeyBox.Tests/PiAgentRunnerTests.cs new file mode 100644 index 00000000..065ac129 --- /dev/null +++ b/tests/CodeyBox.Tests/PiAgentRunnerTests.cs @@ -0,0 +1,282 @@ +using CodeyBox.Agents.Pi; +using CodeyBox.Core; +using CodeyBox.Sandbox; +using Microsoft.Extensions.Logging.Abstractions; + +namespace CodeyBox.Tests; + +/// +/// Unit tests for . Uses the shared +/// CapturingSandbox to inspect the argv and stdin the runner forwards. +/// Argv pins here encode the transport decision verified against pi 0.85.1: +/// --mode json (never raw -p, never --mode rpc), prompt +/// via stdin, --offline, and no approve/trust override. +/// +public sealed class PiAgentRunnerTests +{ + [Fact] + public void Kind_IsPi() + { + Assert.Equal(AgentKind.Pi, new PiAgentRunner().Kind); + } + + [Fact] + public void AgentKind_Pi_RoundTrips() + { + Assert.Equal(AgentKind.Pi, new AgentKind("pi")); + } + + [Fact] + public async Task RunAsync_Argv_UsesModeJsonTransport() + { + var sandbox = new CapturingSandbox(); + var runner = new PiAgentRunner(); + + await runner.RunAsync(sandbox, "/work", "do the thing", credential: null); + + var argv = sandbox.CapturedExec!.Argv.ToList(); + Assert.Equal("pi", argv[0]); + var modeIdx = argv.IndexOf("--mode"); + Assert.True(modeIdx >= 0, "expected --mode flag"); + Assert.Equal("json", argv[modeIdx + 1]); + } + + [Fact] + public async Task RunAsync_Argv_NeverUsesPrintOrRpc() + { + // Raw -p would drop usage/model/error signal; --mode rpc needs a + // bidirectional driver loop this runner does not implement. + var sandbox = new CapturingSandbox(); + var runner = new PiAgentRunner(); + + await runner.RunAsync(sandbox, "/work", "x", credential: null); + + var argv = sandbox.CapturedExec!.Argv.ToList(); + Assert.DoesNotContain(argv, a => a == "-p" || a == "--print"); + Assert.DoesNotContain("rpc", argv); + } + + [Fact] + public async Task RunAsync_Argv_HasNoSessionAndOffline() + { + var sandbox = new CapturingSandbox(); + var runner = new PiAgentRunner(); + + await runner.RunAsync(sandbox, "/work", "x", credential: null); + + var argv = sandbox.CapturedExec!.Argv.ToList(); + Assert.Contains("--no-session", argv); + Assert.Contains("--offline", argv); + } + + [Fact] + public async Task RunAsync_Prompt_TravelsViaStdin_NotArgv() + { + // Linux MAX_ARG_STRLEN is 128 KiB per argv element; rework prompts can + // exceed it. Verified: pi --mode json accepts a stdin-only prompt. + var sandbox = new CapturingSandbox(); + var runner = new PiAgentRunner(); + const string prompt = "implement the widget with extra care"; + + await runner.RunAsync(sandbox, "/work", prompt, credential: null); + + Assert.Equal(prompt, sandbox.CapturedExec!.Stdin); + Assert.DoesNotContain(sandbox.CapturedExec.Argv, a => a.Contains("widget", StringComparison.Ordinal)); + } + + [Fact] + public async Task RunAsync_Argv_NeverOverridesProjectTrust() + { + // Neither --approve nor --no-approve: the sandbox tree is untrusted, + // so pi's ask default (ignore project resources) is the safe posture. + var sandbox = new CapturingSandbox(); + var runner = new PiAgentRunner(); + + await runner.RunAsync(sandbox, "/work", "x", credential: null); + + var argv = sandbox.CapturedExec!.Argv.ToList(); + Assert.DoesNotContain(argv, a => a.StartsWith("--approve", StringComparison.Ordinal)); + } + + [Fact] + public async Task RunAsync_NoModelAnywhere_OmitsModelFlag() + { + var sandbox = new CapturingSandbox(); + var runner = new PiAgentRunner(); + + await runner.RunAsync(sandbox, "/work", "x", credential: null); + + Assert.DoesNotContain("--model", sandbox.CapturedExec!.Argv); + } + + [Fact] + public async Task RunAsync_ExplicitModelId_PassedVerbatim() + { + var sandbox = new CapturingSandbox(); + var runner = new PiAgentRunner(); + + await runner.RunAsync(sandbox, "/work", "x", credential: null, + modelId: "anthropic/claude-haiku-4-5"); + + var argv = sandbox.CapturedExec!.Argv.ToList(); + var modelIdx = argv.IndexOf("--model"); + Assert.True(modelIdx >= 0); + Assert.Equal("anthropic/claude-haiku-4-5", argv[modelIdx + 1]); + } + + [Fact] + public async Task RunAsync_ConfigDefaultModel_UsedWhenNoExplicitModel() + { + var sandbox = new CapturingSandbox(); + var defaults = new AgentDefaultsSnapshot( + new Dictionary(StringComparer.OrdinalIgnoreCase) + { + ["pi"] = "anthropic/claude-haiku-4-5", + }); + var runner = new PiAgentRunner(defaults); + + await runner.RunAsync(sandbox, "/work", "x", credential: null); + + var argv = sandbox.CapturedExec!.Argv.ToList(); + var modelIdx = argv.IndexOf("--model"); + Assert.True(modelIdx >= 0); + Assert.Equal("anthropic/claude-haiku-4-5", argv[modelIdx + 1]); + } + + [Fact] + public async Task DefaultModelId_IsNullWithoutConfig() + { + Assert.Null(new PiAgentRunner().DefaultModelId); + } + + [Theory] + [InlineData("off")] + [InlineData("minimal")] + [InlineData("low")] + [InlineData("medium")] + [InlineData("high")] + [InlineData("xhigh")] + [InlineData("max")] + public async Task RunAsync_KnownThinkingLevel_Emitted(string level) + { + var sandbox = new CapturingSandbox(); + var runner = new PiAgentRunner(); + + await runner.RunAsync(sandbox, "/work", "x", credential: null, reasoningMode: level); + + var argv = sandbox.CapturedExec!.Argv.ToList(); + var idx = argv.IndexOf("--thinking"); + Assert.True(idx >= 0); + Assert.Equal(level, argv[idx + 1]); + } + + [Theory] + [InlineData("ultra")] + [InlineData("high ")] + [InlineData("--thinking")] + public async Task RunAsync_UnknownThinkingLevel_Dropped(string level) + { + // A typo must not fail the CLI invocation; the flag is omitted. + var sandbox = new CapturingSandbox(); + var runner = new PiAgentRunner(); + + await runner.RunAsync(sandbox, "/work", "x", credential: null, reasoningMode: level); + + Assert.DoesNotContain("--thinking", sandbox.CapturedExec!.Argv); + } + + [Fact] + public async Task RunAsync_ExitZeroWithTerminalJsonError_LiftsTerminalDiagnostic() + { + // Verified pi shape: exit 0 with stopReason=error in the event + // stream. Without the lift this would terminal-fail as "produced no + // changes" instead of parking on the quota/auth signal. + const string errorFrame = + """{"type":"message_end","message":{"role":"assistant","provider":"anthropic","model":"claude-haiku-4-5","usage":{"input":0,"output":0,"cacheRead":0,"cacheWrite":0,"totalTokens":0},"stopReason":"error","errorMessage":"401 {\"type\":\"error\",\"error\":{\"type\":\"authentication_error\"}}"}}"""; + var sandbox = new CapturingSandbox( + exitCode: 0, + stdout: "{\"type\":\"session\",\"version\":3}\n" + errorFrame + "\n", + stderr: string.Empty); + var runner = new PiAgentRunner(); + + var result = await runner.RunAsync(sandbox, "/work", "x", credential: null); + + Assert.True(result.Success); + Assert.NotNull(result.TerminalDiagnostic); + Assert.Contains("authentication_error", result.TerminalDiagnostic); + } + + [Fact] + public async Task RunAsync_HealthyRun_LeavesTerminalDiagnosticEmpty() + { + var sandbox = new CapturingSandbox( + exitCode: 0, + stdout: "{\"type\":\"session\",\"version\":3}\n{\"type\":\"agent_end\",\"messages\":[],\"willRetry\":false}\n", + stderr: string.Empty); + var runner = new PiAgentRunner(); + + var result = await runner.RunAsync(sandbox, "/work", "x", credential: null); + + Assert.True(result.Success); + Assert.Null(result.TerminalDiagnostic); + } + + [Fact] + public async Task SupportsStructuredStreamAsync_HelpAdvertisesModeJson_ReturnsTrue() + { + var sandbox = new CapturingSandbox { HelpOutput = "--mode text (default), json, or rpc\n" }; + var runner = new PiAgentRunner(); + + Assert.True(await runner.SupportsStructuredStreamAsync(sandbox)); + } + + [Fact] + public async Task SupportsStructuredStreamAsync_HelpWithoutMode_ReturnsFalse() + { + var sandbox = new CapturingSandbox { HelpOutput = "usage: pi [options]\n" }; + var runner = new PiAgentRunner(); + + Assert.False(await runner.SupportsStructuredStreamAsync(sandbox)); + } + + [Fact] + public void InVmSmokeProbe_BuildsVersionAndHelpSteps() + { + var steps = new PiInVmSmokeProbe().BuildSteps(credential: null); + + Assert.Equal(2, steps.Count); + Assert.Equal(["pi", "--version"], steps[0].Argv); + // The second step asserts --mode support through grep's exit code + // (InVmSmokeStep is exit-code-only by contract). + Assert.Contains("--mode", string.Join(" ", steps[1].Argv)); + } + + [Fact] + public async Task ModelListProbe_ReturnsKnownSeed() + { + var probe = new PiModelListProbe(); + + Assert.Equal(AgentKind.Pi, probe.Kind); + var result = await probe.GetModelListAsync(CancellationToken.None); + + Assert.Null(result.FailureReason); + Assert.Contains("anthropic/claude-haiku-4-5", result.ModelIds); + } + + [Fact] + public void KnownModels_ValidateKnownId_NoWarning() + { + var message = PiKnownModels.ValidateModelIdAgainstProviderList("cls", "anthropic/claude-haiku-4-5", NullLogger.Instance); + + Assert.Null(message); + } + + [Fact] + public void KnownModels_ValidateUnknownId_WarnsButAllows() + { + var message = PiKnownModels.ValidateModelIdAgainstProviderList("cls", "anthropic/claude-zephyr-9", NullLogger.Instance); + + Assert.NotNull(message); + Assert.Contains("claude-zephyr-9", message); + } +} diff --git a/tests/CodeyBox.Tests/PiCostExtractorTests.cs b/tests/CodeyBox.Tests/PiCostExtractorTests.cs new file mode 100644 index 00000000..27b27ca4 --- /dev/null +++ b/tests/CodeyBox.Tests/PiCostExtractorTests.cs @@ -0,0 +1,132 @@ +using CodeyBox.Agents.Pi; +using CodeyBox.Core; + +namespace CodeyBox.Tests; + +/// +/// Tests for . Pi reports cumulative +/// provider-reported usage {input, output, cacheRead, …} plus the +/// dispatch model on every assistant message frame; the extractor keeps the +/// latest usage object as the run total. +/// +public sealed class PiCostExtractorTests +{ + private static readonly PiCostExtractor Extractor = new(); + + [Fact] + public void Kind_IsPi() + { + Assert.Equal(AgentKind.Pi, Extractor.Kind); + } + + [Fact] + public void DefaultPricing_IsNull() + { + // Pi fronts 30+ providers with unrelated per-token economics; no + // single fallback rate is honest. Operators configure per-model rates + // under CodeyBox:AgentPricing (or rely on the bundled pi bucket). + Assert.Null(Extractor.DefaultPricing); + } + + [Fact] + public void NullAndEmpty_ReturnsNull() + { + Assert.Null(Extractor.TryExtract(null, null)); + Assert.Null(Extractor.TryExtract("", "")); + Assert.Null(Extractor.TryExtract(" ", null)); + Assert.Null(Extractor.TryExtract(null, " ")); + } + + [Fact] + public void MessageUsage_ParsesInputOutputAndCacheRead() + { + const string stdout = + "{\"type\":\"message_end\",\"message\":{\"role\":\"assistant\",\"model\":\"claude-haiku-4-5\"," + + "\"usage\":{\"input\":1200,\"output\":340,\"cacheRead\":5600,\"cacheWrite\":10,\"totalTokens\":7140}}}\n"; + + var result = Extractor.TryExtract(stdout, null); + + Assert.NotNull(result); + Assert.Equal(1200, result!.InputTokens); + Assert.Equal(340, result.OutputTokens); + Assert.Equal(5600, result.CachedInputTokens); + Assert.Equal("claude-haiku-4-5", result.ModelId); + } + + [Fact] + public void NestedTurnFrameUsage_ParsedThroughMessageEnvelope() + { + const string stdout = + "{\"type\":\"turn_end\",\"timestamp\":1,\"message\":{\"role\":\"assistant\",\"model\":\"gpt-4o\"," + + "\"usage\":{\"input\":10,\"output\":5,\"cacheRead\":0,\"cacheWrite\":0,\"totalTokens\":15}}}\n"; + + var result = Extractor.TryExtract(stdout, null); + + Assert.NotNull(result); + Assert.Equal(10, result!.InputTokens); + Assert.Equal(5, result.OutputTokens); + Assert.Equal("gpt-4o", result.ModelId); + } + + [Fact] + public void LatestUsageFrame_Wins() + { + // Usage is cumulative per session: the last frame is the run total. + const string stdout = + "{\"type\":\"message_end\",\"message\":{\"model\":\"m\",\"usage\":{\"input\":100,\"output\":10,\"cacheRead\":0,\"cacheWrite\":0,\"totalTokens\":110}}}\n" + + "{\"type\":\"agent_end\",\"messages\":[],\"willRetry\":false,\"message\":{\"model\":\"m\",\"usage\":{\"input\":900,\"output\":90,\"cacheRead\":50,\"cacheWrite\":0,\"totalTokens\":1040}}}\n"; + + var result = Extractor.TryExtract(stdout, null); + + Assert.NotNull(result); + Assert.Equal(900, result!.InputTokens); + Assert.Equal(90, result.OutputTokens); + Assert.Equal(50, result.CachedInputTokens); + } + + [Fact] + public void ZeroUsageFrames_ReturnNull() + { + // Error runs report all-zero usage; there is nothing to attribute. + const string stdout = + "{\"type\":\"message_end\",\"message\":{\"model\":\"claude-haiku-4-5\"," + + "\"usage\":{\"input\":0,\"output\":0,\"cacheRead\":0,\"cacheWrite\":0,\"totalTokens\":0},\"stopReason\":\"error\"}}\n"; + + Assert.Null(Extractor.TryExtract(stdout, null)); + } + + [Fact] + public void UsageOnStderr_Extracted() + { + const string stderr = + "{\"type\":\"message_end\",\"message\":{\"model\":\"m\",\"usage\":{\"input\":7,\"output\":3,\"cacheRead\":0,\"cacheWrite\":0,\"totalTokens\":10}}}\n"; + + var result = Extractor.TryExtract(null, stderr); + + Assert.NotNull(result); + Assert.Equal(7, result!.InputTokens); + } + + [Fact] + public void NonJsonChatter_Skipped() + { + const string stdout = + "some plaintext banner\n" + + "{\"type\":\"message_end\",\"message\":{\"model\":\"m\",\"usage\":{\"input\":11,\"output\":2,\"cacheRead\":0,\"cacheWrite\":0,\"totalTokens\":13}}}\n" + + "{\"broken\": \n"; + + var result = Extractor.TryExtract(stdout, null); + + Assert.NotNull(result); + Assert.Equal(11, result!.InputTokens); + } + + [Fact] + public void NeverThrows_OnHostileInput() + { + const string stdout = "{\"usage\":[1,2,{\"input\":\"x\"}]}\n{\"usage\":{\"input\":-5,\"output\":null}}\n"; + + // Must return normally (null here — no positive counts), never throw. + Assert.Null(Extractor.TryExtract(stdout, stdout)); + } +} diff --git a/tests/CodeyBox.Tests/PiQuotaFailureDetectorTests.cs b/tests/CodeyBox.Tests/PiQuotaFailureDetectorTests.cs new file mode 100644 index 00000000..1fc9f5eb --- /dev/null +++ b/tests/CodeyBox.Tests/PiQuotaFailureDetectorTests.cs @@ -0,0 +1,170 @@ +using CodeyBox.Agents.Pi; +using CodeyBox.Core; +using Microsoft.Extensions.Logging.Abstractions; + +namespace CodeyBox.Tests; + +/// +/// Tests for and . +/// Detector fixtures use the provider-error shapes pi relays verbatim in +/// errorMessage (verified against pi 0.85.1) plus the plaintext +/// pre-session failure line. +/// +public sealed class PiQuotaFailureDetectorTests +{ + private static readonly PiQuotaFailureDetector Detector = new(); + + [Fact] + public void Kind_IsPi() + { + Assert.Equal(AgentKind.Pi, Detector.Kind); + } + + [Fact] + public void NullAndEmpty_ReturnsNull() + { + Assert.Null(Detector.Detect(null, null)); + Assert.Null(Detector.Detect("", "")); + } + + [Fact] + public void LiveAuthErrorFrame_DetectedAsUnauthorized() + { + // Live shape: bogus ANTHROPIC_API_KEY, exit-0 run, error on stdout. + const string stdout = + "{\"type\":\"message_end\",\"message\":{\"stopReason\":\"error\"," + + "\"errorMessage\":\"401 {\\\"type\\\":\\\"error\\\",\\\"error\\\":{\\\"type\\\":\\\"authentication_error\\\",\\\"message\\\":\\\"API key is invalid\\\"}}\"}}"; + + var detection = Detector.Detect(null, stdout); + + Assert.NotNull(detection); + Assert.Equal(QuotaFailureKind.Unauthorized, detection!.Kind); + } + + [Fact] + public void NoApiKeyPlaintext_DetectedAsUnauthorized() + { + var detection = Detector.Detect(null, "No API key found for the selected model.\n"); + + Assert.NotNull(detection); + Assert.Equal(QuotaFailureKind.Unauthorized, detection!.Kind); + } + + [Fact] + public void RateLimitError_DetectedAsRateLimitExceeded() + { + var detection = Detector.Detect( + null, + "{\"type\":\"message_end\",\"message\":{\"stopReason\":\"error\",\"errorMessage\":\"429 rate_limit_exceeded\"}}"); + + Assert.NotNull(detection); + Assert.Equal(QuotaFailureKind.RateLimitExceeded, detection!.Kind); + } + + [Fact] + public void InsufficientQuota_DetectedAsLimitReached() + { + var detection = Detector.Detect( + "Error: insufficient_quota: You exceeded your current quota", + null); + + Assert.NotNull(detection); + Assert.Equal(QuotaFailureKind.LimitReached, detection!.Kind); + } + + [Fact] + public void BareNumber_NotDetected() + { + // Model output citing "429" (retry counts, code under review) must + // not gate dispatch — patterns stay anchored to provider phrasing. + Assert.Null(Detector.Detect(null, "retry 429 times then back off")); + Assert.Null(Detector.Detect("quota", "quota")); + } + + [Fact] + public void HealthyRun_NotDetected() + { + const string stdout = + "{\"type\":\"session\",\"version\":3}\n{\"type\":\"agent_end\",\"messages\":[],\"willRetry\":false}\n"; + + Assert.Null(Detector.Detect(null, stdout)); + } + + [Fact] + public void OperatorExtraPatterns_Appended() + { + var detector = new PiQuotaFailureDetector( + [new QuotaFailurePattern("pi says no", QuotaFailureKind.LimitReached)]); + + var detection = detector.Detect(null, "pi says no today"); + + Assert.NotNull(detection); + Assert.Equal(QuotaFailureKind.LimitReached, detection!.Kind); + } +} + +/// +/// Tests for : credential-presence only, no network. +/// +public sealed class PiSmokeProbeTests +{ + private static PiSmokeProbe NewProbe() => new(NullLogger.Instance); + + private static AgentCredential CredWithKey(string key = "sk-ant-test") => + new(AgentKind.Pi, + new Dictionary { ["ANTHROPIC_API_KEY"] = key }, + new Dictionary()); + + private static AgentCredential EmptyCred() => + new(AgentKind.Pi, + new Dictionary(), + new Dictionary()); + + [Fact] + public async Task Kind_IsPi() + { + Assert.Equal(AgentKind.Pi, NewProbe().Kind); + } + + [Fact] + public async Task CredentialBundleContainsApiKey_ReturnsOk() + { + var result = await NewProbe().SmokeTestAsync(CredWithKey(), CancellationToken.None); + + Assert.True(result.Ok); + Assert.Null(result.FailureReason); + } + + [Fact] + public async Task CredentialBundleContainsApiKey_OkRegardlessOfValueContents() + { + // The probe deliberately does not validate the key — that is the + // runner's responsibility on the first real CLI invocation. + var result = await NewProbe().SmokeTestAsync(CredWithKey("bogus"), CancellationToken.None); + + Assert.True(result.Ok); + } + + [Fact] + public async Task EmptyCredentialBundle_ReturnsFail_WithConfigurationHint() + { + var result = await NewProbe().SmokeTestAsync(EmptyCred(), CancellationToken.None); + + Assert.False(result.Ok); + Assert.NotNull(result.FailureReason); + Assert.Contains("CODEYBOX_PI_API_KEY", result.FailureReason); + } + + [Fact] + public async Task CredentialWithUnrelatedEnvVars_ReturnsFail() + { + var credential = new AgentCredential( + AgentKind.Pi, + new Dictionary { ["UNRELATED"] = "value" }, + new Dictionary()); + + var result = await NewProbe().SmokeTestAsync(credential, CancellationToken.None); + + Assert.False(result.Ok); + } +} diff --git a/tests/CodeyBox.Tests/PiStreamParserTests.cs b/tests/CodeyBox.Tests/PiStreamParserTests.cs new file mode 100644 index 00000000..67fc8513 --- /dev/null +++ b/tests/CodeyBox.Tests/PiStreamParserTests.cs @@ -0,0 +1,114 @@ +using System.Text.Json; +using CodeyBox.Agents.Pi; +using CodeyBox.Core; + +namespace CodeyBox.Tests; + +/// +/// Tests for : pi's --mode json claim +/// vocabulary, pi-named usage mapping, and non-interference with the other +/// registered parsers' claims. +/// +public sealed class PiStreamParserTests +{ + private static Stream StreamOf(string text) + { + var stream = new MemoryStream(); + var writer = new StreamWriter(stream); + writer.Write(text); + writer.Flush(); + stream.Position = 0; + return stream; + } + + [Fact] + public void TryClaim_SessionHeader_Claimed() + { + using var doc = JsonDocument.Parse( + "{\"type\":\"session\",\"version\":3,\"id\":\"s\",\"cwd\":\"/work\"}"); + + Assert.True(new PiStreamParser().TryClaim(doc.RootElement)); + } + + [Fact] + public void TryClaim_SessionWithoutVersionOrCwd_NotClaimed() + { + // A foreign {"type":"session"} line must never be misattributed. + using var doc = JsonDocument.Parse("{\"type\":\"session\"}"); + + Assert.False(new PiStreamParser().TryClaim(doc.RootElement)); + } + + [Theory] + [InlineData("agent_start")] + [InlineData("agent_end")] + [InlineData("agent_settled")] + [InlineData("turn_start")] + [InlineData("turn_end")] + [InlineData("message_start")] + [InlineData("message_update")] + [InlineData("message_end")] + public void TryClaim_LifecycleVerbs_Claimed(string type) + { + using var doc = JsonDocument.Parse($"{{\"type\":\"{type}\"}}"); + + Assert.True(new PiStreamParser().TryClaim(doc.RootElement)); + } + + [Theory] + [InlineData("assistant")] + [InlineData("result")] + [InlineData("tool_use")] + [InlineData("tool_result")] + public void TryClaim_ClaudeVocabulary_NotClaimed(string type) + { + // Claude's parser owns these; pi must not steal Claude streams. + using var doc = JsonDocument.Parse($"{{\"type\":\"{type}\"}}"); + + Assert.False(new PiStreamParser().TryClaim(doc.RootElement)); + } + + [Fact] + public void TryClaim_CodexDottedVocabulary_NotClaimed() + { + using var doc = JsonDocument.Parse("{\"type\":\"turn.started\"}"); + + Assert.False(new PiStreamParser().TryClaim(doc.RootElement)); + } + + [Fact] + public async Task ParseAsync_MapsPiUsageNames() + { + // The shared ParseUsage only knows input_tokens/prompt_tokens; pi + // reports usage:{input,output,cacheRead}. + var parser = new PiStreamParser(); + await using var stream = StreamOf( + "{\"type\":\"session\",\"version\":3,\"id\":\"s\",\"cwd\":\"/work\"}\n" + + "{\"type\":\"message_end\",\"message\":{\"role\":\"assistant\",\"model\":\"m\"," + + "\"usage\":{\"input\":1200,\"output\":340,\"cacheRead\":5600,\"cacheWrite\":10,\"totalTokens\":7140}}}\n" + + "{\"type\":\"agent_end\",\"messages\":[],\"willRetry\":false}\n"); + + var summary = await parser.ParseAsync(stream); + + Assert.Equal(1200, summary.InputTokens); + Assert.Equal(340, summary.OutputTokens); + Assert.Equal(5600, summary.CachedInputTokens); + } + + [Fact] + public async Task ParseAsync_SessionHeaderAndLifecycle_DoNotCorruptSummary() + { + var parser = new PiStreamParser(); + await using var stream = StreamOf( + "{\"type\":\"session\",\"version\":3,\"id\":\"s\",\"cwd\":\"/work\"}\n" + + "{\"type\":\"agent_start\"}\n" + + "{\"type\":\"turn_start\"}\n" + + "{\"type\":\"turn_end\"}\n" + + "{\"type\":\"agent_end\",\"messages\":[],\"willRetry\":false}\n"); + + var summary = await parser.ParseAsync(stream); + + Assert.Equal(0, summary.InputTokens); + Assert.Empty(summary.ToolCalls); + } +} diff --git a/tests/CodeyBox.Tests/PiTerminalDiagnoserTests.cs b/tests/CodeyBox.Tests/PiTerminalDiagnoserTests.cs new file mode 100644 index 00000000..a4909919 --- /dev/null +++ b/tests/CodeyBox.Tests/PiTerminalDiagnoserTests.cs @@ -0,0 +1,92 @@ +using CodeyBox.Agents.Pi; +using CodeyBox.Core; + +namespace CodeyBox.Tests; + +/// +/// Tests for using the terminal shapes +/// verified against pi 0.85.1: exit-0 runs whose only failure signal is the +/// JSON event stream (stopReason: "error" + errorMessage) or the +/// plaintext No API key found pre-session line. +/// +public sealed class PiTerminalDiagnoserTests +{ + [Fact] + public void NullAndEmpty_ReturnsNull() + { + Assert.Null(PiTerminalDiagnoser.TryExtractTerminalError(null)); + Assert.Null(PiTerminalDiagnoser.TryExtractTerminalError("")); + Assert.Null(PiTerminalDiagnoser.TryExtractTerminalError(" ")); + } + + [Fact] + public void HealthyRun_ReturnsNull() + { + const string stdout = + "{\"type\":\"session\",\"version\":3,\"id\":\"s\",\"cwd\":\"/work\"}\n" + + "{\"type\":\"agent_end\",\"messages\":[],\"willRetry\":false}\n"; + + Assert.Null(PiTerminalDiagnoser.TryExtractTerminalError(stdout)); + } + + [Fact] + public void StopReasonError_ExtractsErrorMessage() + { + // Live shape from the pi 0.85.1 auth probe (bogus ANTHROPIC_API_KEY). + const string stdout = + "{\"type\":\"session\",\"version\":3}\n" + + "{\"type\":\"message_end\",\"timestamp\":0,\"message\":{\"role\":\"assistant\",\"provider\":\"anthropic\",\"model\":\"claude-haiku-4-5\"," + + "\"usage\":{\"input\":0,\"output\":0,\"cacheRead\":0,\"cacheWrite\":0,\"totalTokens\":0,\"cost\":{\"input\":0,\"output\":0,\"cacheRead\":0,\"cacheWrite\":0,\"total\":0}}," + + "\"stopReason\":\"error\",\"errorMessage\":\"401 {\\\"type\\\":\\\"error\\\",\\\"error\\\":{\\\"type\\\":\\\"authentication_error\\\"}}\",\"cost\":0}}\n"; + + var result = PiTerminalDiagnoser.TryExtractTerminalError(stdout); + + Assert.NotNull(result); + Assert.Contains("authentication_error", result); + } + + [Fact] + public void NoApiKeyPlaintextLine_Extracted() + { + const string stdout = + "{\"type\":\"session\",\"version\":3}\n" + + "No API key found for the selected model.\n"; + + var result = PiTerminalDiagnoser.TryExtractTerminalError(stdout); + + Assert.NotNull(result); + Assert.Contains("No API key found", result); + } + + [Fact] + public void NonErrorStopReason_Ignored() + { + const string stdout = + "{\"type\":\"message_end\",\"message\":{\"role\":\"assistant\",\"stopReason\":\"stop\"}}\n"; + + Assert.Null(PiTerminalDiagnoser.TryExtractTerminalError(stdout)); + } + + [Fact] + public void MalformedLines_Skipped_ErrorStillFound() + { + const string stdout = + "not json at all\n" + + "{\"type\":\"message_end\",\"message\":{\"stopReason\":\"error\",\"errorMessage\":\"boom\"}}\n" + + "{\"truncated\": true\n"; + + Assert.Equal("boom", PiTerminalDiagnoser.TryExtractTerminalError(stdout)); + } + + [Fact] + public void LongErrorMessage_Truncated() + { + var longError = new string('e', PiTerminalDiagnoser.MaxDiagnosticChars + 100); + var stdout = "{\"type\":\"message_end\",\"message\":{\"stopReason\":\"error\",\"errorMessage\":\"" + longError + "\"}}\n"; + + var result = PiTerminalDiagnoser.TryExtractTerminalError(stdout); + + Assert.NotNull(result); + Assert.True(result!.Length <= PiTerminalDiagnoser.MaxDiagnosticChars + 1); + } +}