Evaluate + add caveman-code as a runner agent

CodeyBox.slnx

@@ -11,6 +11,7 @@
     <Project Path="src/CodeyBox.Agents.Antigravity/CodeyBox.Agents.Antigravity.csproj" />
     <Project Path="src/CodeyBox.Agents.Crock/CodeyBox.Agents.Crock.csproj" />
     <Project Path="src/CodeyBox.Agents.Pi/CodeyBox.Agents.Pi.csproj" />
+    <Project Path="src/CodeyBox.Agents.CavemanCode/CodeyBox.Agents.CavemanCode.csproj" />
     <Project Path="src/CodeyBox.Agents/CodeyBox.Agents.csproj" />
     <Project Path="src/CodeyBox.AdminSeed/CodeyBox.AdminSeed.csproj" />
     <Project Path="src/CodeyBox.Api/CodeyBox.Api.csproj" />

docs/concepts/agents.md

@@ -49,6 +49,7 @@ the most common cause of fresh-class dispatch failures.
 | `gemini`  | `npm install -g @google/gemini-cli` | `ReasoningMode` is **not** wired into argv — Gemini's reasoning level is encoded in `ModelId` (pick a `gemini-3-*-preview` model for HIGH). See [Gemini quirks](../reference/agent-quirks.md#google-gemini-cli-googlegemini-cli). |
 | `cursor`  | `curl -fsSL https://cursor.com/install \| bash` | Installs as `agent` (not `cursor-agent`). See [Cursor quirks](../reference/agent-quirks.md#cursor-cli-agent). |
 | `opencode` | `curl -fsSL https://opencode.ai/install \| bash` | Plaintext stdout only — no structured stream. |
+| `caveman` | `npm install -g @juliusbrussee/caveman-code` | Invoked as `caveman-code` (the unambiguous alias — the successor `caveman wrap` package ships a colliding `caveman` binary). Needs Node.js 20+ on the image. Plaintext stdout only — no structured stream. |
 | `antigravity` | *operator-supplied — stage the `agy` binary on the host and ship it via `CodeyBox:MultipassExecutableProvisions` or `CodeyBox:Incus:ExecutableProvisions`, matching the selected provider* (see [Antigravity quirks](../reference/agent-quirks.md#google-antigravity-cli-agy)). Do not use `curl -fsSL https://antigravity.google/cli/install.sh \| bash`: that URL serves the landing page, not a script, and piping HTML into `bash` fails silently when the runcmd ends with `\|\| true`. | Installs the proprietary `agy` CLI on the non-login sandbox PATH. Multi-model gateway — each gateway model id is a separate quota bucket. Configure each accepted model as its own `AgentClass` member; the router gates per-model via the existing `(AgentKind, ModelId)` exhaustion key. |
 | `pi` | `npm install -g --ignore-scripts @earendil-works/pi-coding-agent` | MIT-licensed; needs Node.js on the image. `--ignore-scripts` skips npm lifecycle scripts during install. See [Pi quirks](../reference/agent-quirks.md#pi-coding-agent-pi). |
 

docs/reference/agent-quirks.md

@@ -751,3 +751,90 @@ frame and the bare `message.model` id (pi strips the `provider/` qualifier in
 `pi` bucket for the shipped Anthropic-backed member; operators fronting other
 providers add that provider's list prices there (or under
 `CodeyBox:AgentPricing`) keyed by the bare model id.
+
+### Caveman-code CLI (`caveman-code`)
+
+Caveman-code (`github.com/JuliusBrussee/caveman-code`, npm
+`@juliusbrussee/caveman-code`, **MIT**) is a standalone terminal coding agent
+in the pi-mono family whose pitch is token compression (~2× fewer tokens than
+Codex CLI on its published 25-task MicroBench: 524k vs 1,010k fresh tokens,
+14/25 vs 15/25 passes, gpt-5.5 xhigh). All CLI behaviour below was verified
+live against 0.65.2 (npm install + `--help` + unauthenticated dispatch +
+`--list-models` with placeholder keys).
+
+**Status: frozen upstream.** The author froze this repo in August 2026; active
+work moved to the `caveman wrap` successor, whose `caveman` binary **shadows**
+this package's primary alias — uninstall one before installing the other. The
+runner therefore invokes the unambiguous `caveman-code` alias (same CLI, no
+collision), and the shipped `frontier-coding` member carries no `sensitive`
+capability: same-model, same-vendor spend as the codex member, but a frozen
+upstream gets conservative clearance until an operator opts in. Expect no
+upstream fixes; treat CLI-flag drift as freeze, not breakage.
+
+**Install in the sandbox image** — needs Node.js 20+:
+
+```sh
+npm install -g @juliusbrussee/caveman-code
+```
+
+**Non-interactive invocation.** `caveman-code -p` ("print mode: process
+prompt and exit") with the prompt on **stdin** — a piped prompt with no
+positional reaches agent init, which keeps large rework prompts under the
+128 KiB MAX_ARG_STRLEN ceiling. Do NOT switch to `caveman-code exec --json`
+for large prompts: the exec subcommand is dispatched before stdin is read, so
+its prompt must ride argv (and it accepts no `--provider`/`--thinking`). Text
+mode prints only the final assistant text; errors go to stderr (exit 1) —
+except the missing-key message, which prints to **stdout with exit 0**, so a
+keyless dispatch looks "successful" unless the smoke gate benches the agent
+first (it does — see below).
+
+**Authentication — BYOK API keys only.** The CLI reads provider keys from the
+environment (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, … —
+full list in `CavemanCodeAgentRunner.CredentialEnvironmentVariables`). OAuth
+(`/login`, OS-keychain tokens) has no headless path and is unsupported. The
+runner never passes `--api-key` (secrets must not ride argv). Operator
+wiring: set `CODEYBOX_CAVEMAN_ANTHROPIC_API_KEY` /
+`CODEYBOX_CAVEMAN_OPENAI_API_KEY` / `CODEYBOX_CAVEMAN_GEMINI_API_KEY` /
+`CODEYBOX_CAVEMAN_OPENROUTER_API_KEY` on the host (or inject the conventional
+provider vars directly — both are mapped through). Endpoint-style providers
+(Azure OpenAI, AWS Bedrock) need companion config beyond a bare key and are
+not covered; per-instance member `CredentialReference` tokens are not mapped
+for this kind (the sandbox var is provider-dependent), so multi-key setups
+use one member per provider key. The key never appears in any log line.
+
+**Default model.** Shipped default is `openai/gpt-5.5` (both
+`CodeyBox:AgentDefaults:caveman` and the `frontier-coding` member): the
+provider-prefixed form pins the provider so `--model` resolves without
+`--provider`, and it is the exact configuration the upstream bench measured.
+`gpt-5.5`, `claude-opus-4-6`, `claude-opus-4-7`, and `claude-sonnet-4-6` are
+confirmed present in the 0.65.2 registry via `caveman-code --list-models`.
+
+**Reasoning effort.** `--thinking off|minimal|low|medium|high|xhigh`
+(verified). The runner forwards `ReasoningMode` only when it matches that set
+(lowercased); anything else is dropped rather than forwarded. Pin thinking
+via member `ReasoningMode`, not a `:suffix` on `ModelId` — a suffixed id will
+not match the model-list probe and trips a startup warning.
+
+**Billing — pay-per-token (`PayPerApi`).** Spend bills to the operator's own
+provider accounts, so the orchestrator never waits on quota for this member.
+Rates live in `agent-pricing-defaults.json` under the `caveman` bucket in both
+bare and provider-prefixed key forms, hot-reloadable under
+`CodeyBox:AgentPricing`. Cost extraction is JSON-envelope-only (caveman
+camelCase usage, Anthropic/OpenAI usage objects): plain-text `-p` runs emit
+no counts, and prose token mentions are never matched, so a normal dispatch
+attributes zero until a structured envelope appears.
+
+**Quota probe.** None — BYOK keys expose no remaining-credit meter (same
+position as opencode). Availability is covered by the credential-presence host
+smoke probe plus the in-VM `caveman-code --list-models` check; transient
+provider refusals (shared 429 shapes, `overloaded_error`, missing-key output)
+are classified by `CavemanCodeQuotaFailureDetector` into rate-limit backoff /
+auth-error paths.
+
+**Model-list probe.** Runs `caveman-code --list-models` on the API host and
+parses the provider table (emits both `provider/model` and bare `model` ids
+so either member spelling validates). The registry answers offline from env
+keys alone, but with no key visible it prints `No models available…` at exit
+0 — the probe treats zero parsed ids as failure. Set
+`CODEYBOX_CAVEMANCODE_BINARY` to override the binary path. When the CLI is
+missing or keyless on the host, validation is skipped with a warning.

src/CodeyBox.Agents.CavemanCode/CavemanCodeAgentRunner.cs

@@ -0,0 +1,157 @@
+using CodeyBox.Agents;
+using CodeyBox.Core;
+using CodeyBox.Sandbox;
+
+namespace CodeyBox.Agents.CavemanCode;
+
+/// <summary>
+/// Drives the <c>caveman-code</c> CLI
+/// (<c>github.com/JuliusBrussee/caveman-code</c>, npm
+/// <c>@juliusbrussee/caveman-code</c>, MIT) in non-interactive print mode.
+///
+/// <para>Invocation is <c>caveman-code -p</c> with the prompt on stdin —
+/// verified against 0.65.2: a piped prompt with no positional message
+/// reaches agent init (it fails at auth, not arg parsing), which dodges the
+/// 128 KiB MAX_ARG_STRLEN ceiling rework prompts can blow through, mirroring
+/// the Gemini runner. <c>--mode json</c> is deliberately NOT used: print-mode
+/// JSON emits the CLI's internal (unfrozen) session events, while the frozen
+/// <c>exec --json</c> stream cannot take the prompt on stdin (the exec
+/// subcommand is dispatched before stdin is read, so large prompts would have
+/// to ride argv). Stdout stays the final assistant text, exactly like the
+/// opencode runner.</para>
+///
+/// <para>Auth is BYOK API keys only: the CLI reads provider keys from the
+/// environment (<c>ANTHROPIC_API_KEY</c>, <c>OPENAI_API_KEY</c>,
+/// <c>GEMINI_API_KEY</c>, … — full list in
+/// <see cref="CredentialEnvironmentVariables"/>). OAuth (<c>/login</c>)
+/// stores tokens in the OS keychain, which has no headless path, so it is
+/// not supported here. Keys arrive via the sandbox environment (see
+/// <see cref="DirectCredentialEnvironmentVariables"/>) — the runner never
+/// passes <c>--api-key</c> because secrets must not ride argv.</para>
+///
+/// <para>Status caveat: upstream froze this repo in August 2026 (active work
+/// moved to the <c>caveman wrap</c> successor, whose <c>caveman</c> binary
+/// shadows this package's primary alias). The runner invokes the unambiguous
+/// <c>caveman-code</c> alias so both packages can coexist on a host. Default
+/// model <c>openai/gpt-5.5</c> is the configuration the upstream 25-task
+/// MicroBench measured (1.93x fewer tokens than Codex CLI, 14/25 vs 15/25
+/// passes, gpt-5.5 xhigh).</para>
+/// </summary>
+public sealed class CavemanCodeAgentRunner : CliAgentRunnerBase, IAgentDefaultModelProvider
+{
+    /// <summary>
+    /// Default CLI binary inside the sandbox. The npm package installs two
+    /// aliases (<c>caveman</c> primary, <c>caveman-code</c>); this runner uses
+    /// the long alias because the successor <c>caveman wrap</c> package
+    /// installs a colliding <c>caveman</c> binary (upstream README warning).
+    /// Shared with <see cref="CavemanCodeInVmSmokeProbe"/> so the smoke check
+    /// and the real runner always invoke the same binary.
+    /// </summary>
+    public const string DefaultBinary = "caveman-code";
+
+    /// <summary>
+    /// Provider API-key variables the CLI reads from its environment,
+    /// verified against <c>caveman-code --help</c> (0.65.2). Endpoint-style
+    /// providers (Azure OpenAI, AWS Bedrock) need companion config beyond a
+    /// bare key and are intentionally not covered — see agent-quirks.
+    /// </summary>
+    public static readonly IReadOnlyList<string> CredentialEnvironmentVariables =
+    [
+        "ANTHROPIC_API_KEY",
+        "OPENAI_API_KEY",
+        "GEMINI_API_KEY",
+        "GOOGLE_API_KEY",
+        "GROQ_API_KEY",
+        "CEREBRAS_API_KEY",
+        "XAI_API_KEY",
+        "OPENROUTER_API_KEY",
+        "MISTRAL_API_KEY",
+        "MINIMAX_API_KEY",
+        "KIMI_API_KEY",
+        "ZAI_API_KEY",
+        "DEEPSEEK_API_KEY",
+        "OPENCODE_API_KEY",
+        "AI_GATEWAY_API_KEY",
+    ];
+
+    /// <summary>
+    /// Thinking levels accepted by <c>--thinking</c>, verified against
+    /// <c>caveman-code --help</c> (0.65.2). A <c>ReasoningMode</c> outside
+    /// this set is dropped rather than forwarded: the CLI degrades an
+    /// unknown level to a startup warning, and a misspelled routing knob
+    /// must not change dispatch behaviour.
+    /// </summary>
+    public static readonly IReadOnlySet<string> ValidThinkingLevels =
+        new HashSet<string>(StringComparer.OrdinalIgnoreCase)
+        {
+            "off", "minimal", "low", "medium", "high", "xhigh",
+        };
+
+    private readonly AgentDefaultsSnapshot? _defaults;
+
+    public CavemanCodeAgentRunner() : this(defaults: null) { }
+
+    public CavemanCodeAgentRunner(AgentDefaultsSnapshot? defaults)
+    {
+        _defaults = defaults;
+    }
+
+    public override AgentKind Kind => AgentKind.CavemanCode;
+
+    /// <summary>Path to the CLI binary inside the sandbox. Defaults to <see cref="DefaultBinary"/>.</summary>
+    public string Binary { get; init; } = DefaultBinary;
+
+    /// <summary>
+    /// Default model passed to <c>--model</c> when the agent-class member
+    /// does not override it. Sourced live from <see cref="AgentDefaultsSnapshot"/>.
+    /// </summary>
+    public string? DefaultModelId => _defaults?.GetDefault(Kind.Value);
+
+    protected override IReadOnlyList<string> ScratchpadHomeDirectories => [".cave"];
+
+    protected override IReadOnlyList<string> DirectCredentialEnvironmentVariables =>
+        CredentialEnvironmentVariables.ToArray();
+
+    protected override string PreemptProcessPattern => Binary;
+
+    /// <summary>
+    /// Builds the <c>caveman-code -p</c> argv. The
+    /// <paramref name="captureStructuredStream"/> parameter is currently
+    /// discarded — print-mode <c>--mode json</c> emits the CLI's internal
+    /// unfrozen session events, and the frozen <c>exec --json</c> stream
+    /// cannot take the prompt on stdin. The runner does not implement
+    /// <see cref="IStructuredStreamAgentRunner"/>; callers requesting
+    /// structured capture get plain stdout/stderr back.
+    /// </summary>
+    protected override AgentInvocation BuildInvocation(
+        string prompt,
+        AgentCredential? credential,
+        string? modelId = null,
+        string? reasoningMode = null,
+        bool captureStructuredStream = false)
+    {
+        // `caveman-code -p` reads the prompt appended to stdin (verified:
+        // `echo ... | caveman-code -p` with no positional reaches auth init).
+        // Stdin keeps large rework prompts under MAX_ARG_STRLEN.
+        var argv = new List<string> { Binary, "-p" };
+
+        var effectiveModel = !string.IsNullOrEmpty(modelId) ? modelId : DefaultModelId;
+        if (!string.IsNullOrEmpty(effectiveModel))
+        {
+            argv.Add("--model");
+            argv.Add(effectiveModel);
+        }
+
+        // Reasoning effort maps 1:1 onto --thinking; the valid set is
+        // verified against --help so only known levels are forwarded.
+        if (!string.IsNullOrEmpty(reasoningMode)
+            && ValidThinkingLevels.Contains(reasoningMode))
+        {
+            argv.Add("--thinking");
+            argv.Add(reasoningMode.ToLowerInvariant());
+        }
+
+        _ = captureStructuredStream;
+        return new AgentInvocation(argv, Stdin: prompt);
+    }
+}