diff --git a/CodeyBox.slnx b/CodeyBox.slnx index 89a507e1f..65f5cb448 100644 --- a/CodeyBox.slnx +++ b/CodeyBox.slnx @@ -11,6 +11,7 @@ + diff --git a/docs/concepts/agents.md b/docs/concepts/agents.md index 9ebb7c4c9..f28f5935b 100644 --- a/docs/concepts/agents.md +++ b/docs/concepts/agents.md @@ -49,6 +49,7 @@ the most common cause of fresh-class dispatch failures. | `gemini` | `npm install -g @google/gemini-cli` | `ReasoningMode` is **not** wired into argv — Gemini's reasoning level is encoded in `ModelId` (pick a `gemini-3-*-preview` model for HIGH). See [Gemini quirks](../reference/agent-quirks.md#google-gemini-cli-googlegemini-cli). | | `cursor` | `curl -fsSL https://cursor.com/install \| bash` | Installs as `agent` (not `cursor-agent`). See [Cursor quirks](../reference/agent-quirks.md#cursor-cli-agent). | | `opencode` | `curl -fsSL https://opencode.ai/install \| bash` | Plaintext stdout only — no structured stream. | +| `caveman` | `npm install -g @juliusbrussee/caveman-code` | Invoked as `caveman-code` (the unambiguous alias — the successor `caveman wrap` package ships a colliding `caveman` binary). Needs Node.js 20+ on the image. Plaintext stdout only — no structured stream. | | `antigravity` | *operator-supplied — stage the `agy` binary on the host and ship it via `CodeyBox:MultipassExecutableProvisions` or `CodeyBox:Incus:ExecutableProvisions`, matching the selected provider* (see [Antigravity quirks](../reference/agent-quirks.md#google-antigravity-cli-agy)). Do not use `curl -fsSL https://antigravity.google/cli/install.sh \| bash`: that URL serves the landing page, not a script, and piping HTML into `bash` fails silently when the runcmd ends with `\|\| true`. | Installs the proprietary `agy` CLI on the non-login sandbox PATH. Multi-model gateway — each gateway model id is a separate quota bucket. Configure each accepted model as its own `AgentClass` member; the router gates per-model via the existing `(AgentKind, ModelId)` exhaustion key. | | `pi` | `npm install -g --ignore-scripts @earendil-works/pi-coding-agent` | MIT-licensed; needs Node.js on the image. `--ignore-scripts` skips npm lifecycle scripts during install. See [Pi quirks](../reference/agent-quirks.md#pi-coding-agent-pi). | diff --git a/docs/reference/agent-quirks.md b/docs/reference/agent-quirks.md index 3bad6ba71..807e3a7ce 100644 --- a/docs/reference/agent-quirks.md +++ b/docs/reference/agent-quirks.md @@ -751,3 +751,90 @@ frame and the bare `message.model` id (pi strips the `provider/` qualifier in `pi` bucket for the shipped Anthropic-backed member; operators fronting other providers add that provider's list prices there (or under `CodeyBox:AgentPricing`) keyed by the bare model id. + +### Caveman-code CLI (`caveman-code`) + +Caveman-code (`github.com/JuliusBrussee/caveman-code`, npm +`@juliusbrussee/caveman-code`, **MIT**) is a standalone terminal coding agent +in the pi-mono family whose pitch is token compression (~2× fewer tokens than +Codex CLI on its published 25-task MicroBench: 524k vs 1,010k fresh tokens, +14/25 vs 15/25 passes, gpt-5.5 xhigh). All CLI behaviour below was verified +live against 0.65.2 (npm install + `--help` + unauthenticated dispatch + +`--list-models` with placeholder keys). + +**Status: frozen upstream.** The author froze this repo in August 2026; active +work moved to the `caveman wrap` successor, whose `caveman` binary **shadows** +this package's primary alias — uninstall one before installing the other. The +runner therefore invokes the unambiguous `caveman-code` alias (same CLI, no +collision), and the shipped `frontier-coding` member carries no `sensitive` +capability: same-model, same-vendor spend as the codex member, but a frozen +upstream gets conservative clearance until an operator opts in. Expect no +upstream fixes; treat CLI-flag drift as freeze, not breakage. + +**Install in the sandbox image** — needs Node.js 20+: + +```sh +npm install -g @juliusbrussee/caveman-code +``` + +**Non-interactive invocation.** `caveman-code -p` ("print mode: process +prompt and exit") with the prompt on **stdin** — a piped prompt with no +positional reaches agent init, which keeps large rework prompts under the +128 KiB MAX_ARG_STRLEN ceiling. Do NOT switch to `caveman-code exec --json` +for large prompts: the exec subcommand is dispatched before stdin is read, so +its prompt must ride argv (and it accepts no `--provider`/`--thinking`). Text +mode prints only the final assistant text; errors go to stderr (exit 1) — +except the missing-key message, which prints to **stdout with exit 0**, so a +keyless dispatch looks "successful" unless the smoke gate benches the agent +first (it does — see below). + +**Authentication — BYOK API keys only.** The CLI reads provider keys from the +environment (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, … — +full list in `CavemanCodeAgentRunner.CredentialEnvironmentVariables`). OAuth +(`/login`, OS-keychain tokens) has no headless path and is unsupported. The +runner never passes `--api-key` (secrets must not ride argv). Operator +wiring: set `CODEYBOX_CAVEMAN_ANTHROPIC_API_KEY` / +`CODEYBOX_CAVEMAN_OPENAI_API_KEY` / `CODEYBOX_CAVEMAN_GEMINI_API_KEY` / +`CODEYBOX_CAVEMAN_OPENROUTER_API_KEY` on the host (or inject the conventional +provider vars directly — both are mapped through). Endpoint-style providers +(Azure OpenAI, AWS Bedrock) need companion config beyond a bare key and are +not covered; per-instance member `CredentialReference` tokens are not mapped +for this kind (the sandbox var is provider-dependent), so multi-key setups +use one member per provider key. The key never appears in any log line. + +**Default model.** Shipped default is `openai/gpt-5.5` (both +`CodeyBox:AgentDefaults:caveman` and the `frontier-coding` member): the +provider-prefixed form pins the provider so `--model` resolves without +`--provider`, and it is the exact configuration the upstream bench measured. +`gpt-5.5`, `claude-opus-4-6`, `claude-opus-4-7`, and `claude-sonnet-4-6` are +confirmed present in the 0.65.2 registry via `caveman-code --list-models`. + +**Reasoning effort.** `--thinking off|minimal|low|medium|high|xhigh` +(verified). The runner forwards `ReasoningMode` only when it matches that set +(lowercased); anything else is dropped rather than forwarded. Pin thinking +via member `ReasoningMode`, not a `:suffix` on `ModelId` — a suffixed id will +not match the model-list probe and trips a startup warning. + +**Billing — pay-per-token (`PayPerApi`).** Spend bills to the operator's own +provider accounts, so the orchestrator never waits on quota for this member. +Rates live in `agent-pricing-defaults.json` under the `caveman` bucket in both +bare and provider-prefixed key forms, hot-reloadable under +`CodeyBox:AgentPricing`. Cost extraction is JSON-envelope-only (caveman +camelCase usage, Anthropic/OpenAI usage objects): plain-text `-p` runs emit +no counts, and prose token mentions are never matched, so a normal dispatch +attributes zero until a structured envelope appears. + +**Quota probe.** None — BYOK keys expose no remaining-credit meter (same +position as opencode). Availability is covered by the credential-presence host +smoke probe plus the in-VM `caveman-code --list-models` check; transient +provider refusals (shared 429 shapes, `overloaded_error`, missing-key output) +are classified by `CavemanCodeQuotaFailureDetector` into rate-limit backoff / +auth-error paths. + +**Model-list probe.** Runs `caveman-code --list-models` on the API host and +parses the provider table (emits both `provider/model` and bare `model` ids +so either member spelling validates). The registry answers offline from env +keys alone, but with no key visible it prints `No models available…` at exit +0 — the probe treats zero parsed ids as failure. Set +`CODEYBOX_CAVEMANCODE_BINARY` to override the binary path. When the CLI is +missing or keyless on the host, validation is skipped with a warning. diff --git a/src/CodeyBox.Agents.CavemanCode/CavemanCodeAgentRunner.cs b/src/CodeyBox.Agents.CavemanCode/CavemanCodeAgentRunner.cs new file mode 100644 index 000000000..011bb1bb5 --- /dev/null +++ b/src/CodeyBox.Agents.CavemanCode/CavemanCodeAgentRunner.cs @@ -0,0 +1,157 @@ +using CodeyBox.Agents; +using CodeyBox.Core; +using CodeyBox.Sandbox; + +namespace CodeyBox.Agents.CavemanCode; + +/// +/// Drives the caveman-code CLI +/// (github.com/JuliusBrussee/caveman-code, npm +/// @juliusbrussee/caveman-code, MIT) in non-interactive print mode. +/// +/// Invocation is caveman-code -p with the prompt on stdin — +/// verified against 0.65.2: a piped prompt with no positional message +/// reaches agent init (it fails at auth, not arg parsing), which dodges the +/// 128 KiB MAX_ARG_STRLEN ceiling rework prompts can blow through, mirroring +/// the Gemini runner. --mode json is deliberately NOT used: print-mode +/// JSON emits the CLI's internal (unfrozen) session events, while the frozen +/// exec --json stream cannot take the prompt on stdin (the exec +/// subcommand is dispatched before stdin is read, so large prompts would have +/// to ride argv). Stdout stays the final assistant text, exactly like the +/// opencode runner. +/// +/// Auth is BYOK API keys only: the CLI reads provider keys from the +/// environment (ANTHROPIC_API_KEY, OPENAI_API_KEY, +/// GEMINI_API_KEY, … — full list in +/// ). OAuth (/login) +/// stores tokens in the OS keychain, which has no headless path, so it is +/// not supported here. Keys arrive via the sandbox environment (see +/// ) — the runner never +/// passes --api-key because secrets must not ride argv. +/// +/// Status caveat: upstream froze this repo in August 2026 (active work +/// moved to the caveman wrap successor, whose caveman binary +/// shadows this package's primary alias). The runner invokes the unambiguous +/// caveman-code alias so both packages can coexist on a host. Default +/// model openai/gpt-5.5 is the configuration the upstream 25-task +/// MicroBench measured (1.93x fewer tokens than Codex CLI, 14/25 vs 15/25 +/// passes, gpt-5.5 xhigh). +/// +public sealed class CavemanCodeAgentRunner : CliAgentRunnerBase, IAgentDefaultModelProvider +{ + /// + /// Default CLI binary inside the sandbox. The npm package installs two + /// aliases (caveman primary, caveman-code); this runner uses + /// the long alias because the successor caveman wrap package + /// installs a colliding caveman binary (upstream README warning). + /// Shared with so the smoke check + /// and the real runner always invoke the same binary. + /// + public const string DefaultBinary = "caveman-code"; + + /// + /// Provider API-key variables the CLI reads from its environment, + /// verified against caveman-code --help (0.65.2). Endpoint-style + /// providers (Azure OpenAI, AWS Bedrock) need companion config beyond a + /// bare key and are intentionally not covered — see agent-quirks. + /// + public static readonly IReadOnlyList CredentialEnvironmentVariables = + [ + "ANTHROPIC_API_KEY", + "OPENAI_API_KEY", + "GEMINI_API_KEY", + "GOOGLE_API_KEY", + "GROQ_API_KEY", + "CEREBRAS_API_KEY", + "XAI_API_KEY", + "OPENROUTER_API_KEY", + "MISTRAL_API_KEY", + "MINIMAX_API_KEY", + "KIMI_API_KEY", + "ZAI_API_KEY", + "DEEPSEEK_API_KEY", + "OPENCODE_API_KEY", + "AI_GATEWAY_API_KEY", + ]; + + /// + /// Thinking levels accepted by --thinking, verified against + /// caveman-code --help (0.65.2). A ReasoningMode outside + /// this set is dropped rather than forwarded: the CLI degrades an + /// unknown level to a startup warning, and a misspelled routing knob + /// must not change dispatch behaviour. + /// + public static readonly IReadOnlySet ValidThinkingLevels = + new HashSet(StringComparer.OrdinalIgnoreCase) + { + "off", "minimal", "low", "medium", "high", "xhigh", + }; + + private readonly AgentDefaultsSnapshot? _defaults; + + public CavemanCodeAgentRunner() : this(defaults: null) { } + + public CavemanCodeAgentRunner(AgentDefaultsSnapshot? defaults) + { + _defaults = defaults; + } + + public override AgentKind Kind => AgentKind.CavemanCode; + + /// Path to the CLI binary inside the sandbox. Defaults to . + public string Binary { get; init; } = DefaultBinary; + + /// + /// Default model passed to --model when the agent-class member + /// does not override it. Sourced live from . + /// + public string? DefaultModelId => _defaults?.GetDefault(Kind.Value); + + protected override IReadOnlyList ScratchpadHomeDirectories => [".cave"]; + + protected override IReadOnlyList DirectCredentialEnvironmentVariables => + CredentialEnvironmentVariables.ToArray(); + + protected override string PreemptProcessPattern => Binary; + + /// + /// Builds the caveman-code -p argv. The + /// parameter is currently + /// discarded — print-mode --mode json emits the CLI's internal + /// unfrozen session events, and the frozen exec --json stream + /// cannot take the prompt on stdin. The runner does not implement + /// ; callers requesting + /// structured capture get plain stdout/stderr back. + /// + protected override AgentInvocation BuildInvocation( + string prompt, + AgentCredential? credential, + string? modelId = null, + string? reasoningMode = null, + bool captureStructuredStream = false) + { + // `caveman-code -p` reads the prompt appended to stdin (verified: + // `echo ... | caveman-code -p` with no positional reaches auth init). + // Stdin keeps large rework prompts under MAX_ARG_STRLEN. + var argv = new List { Binary, "-p" }; + + var effectiveModel = !string.IsNullOrEmpty(modelId) ? modelId : DefaultModelId; + if (!string.IsNullOrEmpty(effectiveModel)) + { + argv.Add("--model"); + argv.Add(effectiveModel); + } + + // Reasoning effort maps 1:1 onto --thinking; the valid set is + // verified against --help so only known levels are forwarded. + if (!string.IsNullOrEmpty(reasoningMode) + && ValidThinkingLevels.Contains(reasoningMode)) + { + argv.Add("--thinking"); + argv.Add(reasoningMode.ToLowerInvariant()); + } + + _ = captureStructuredStream; + return new AgentInvocation(argv, Stdin: prompt); + } +} diff --git a/src/CodeyBox.Agents.CavemanCode/CavemanCodeCostExtractor.cs b/src/CodeyBox.Agents.CavemanCode/CavemanCodeCostExtractor.cs new file mode 100644 index 000000000..7898e3adb --- /dev/null +++ b/src/CodeyBox.Agents.CavemanCode/CavemanCodeCostExtractor.cs @@ -0,0 +1,207 @@ +using System.Text.Json; +using CodeyBox.Agents; +using CodeyBox.Core; + +namespace CodeyBox.Agents.CavemanCode; + +/// +/// Best-effort token-count extractor for caveman-code CLI output. +/// +/// Plain-text -p runs emit only the final assistant prose, so a +/// normal dispatch yields no counts and this returns null (zero-cost row, +/// same as opencode when its CLI stays quiet). What IS parsed is +/// machine-shaped JSON only — never prose: +/// +/// caveman camelCase usage objects +/// (inputTokens/outputTokens), the field names the CLI's own +/// exec event translator reads (shipped dist/modes/exec/event-stream.js, +/// 0.65.2), at root.usage or root.message.usage; +/// Anthropic snake_case envelopes via the shared +/// (same shape the +/// Claude extractor consumes); +/// OpenAI prompt_tokens/completion_tokens objects +/// including prompt_tokens_details.cached_tokens. +/// +/// +/// Prose token mentions (e.g. an agent writing "10 input tokens" in its +/// answer) are deliberately NOT matched: with no usage footer in text mode, +/// any such match could only come from the model's own words and would +/// fabricate spend. Totals accumulate across usage-bearing lines, matching +/// the CLI's per-assistant-message cost events (each +/// message.assistant carries its own turn cost; the session total is +/// their sum). Counters are clamped non-negative with saturating adds — +/// agent stdout is a less-trusted input to budget gates. +/// +/// No is shipped: caveman-code fronts many +/// providers with different per-token economics. Per-model rates ship in +/// agent-pricing-defaults.json under the caveman bucket (both +/// bare gpt-5.5 and provider-prefixed openai/gpt-5.5 key forms, +/// since the CLI reports either depending on how --model was given) +/// and stay hot-reloadable under CodeyBox:AgentPricing. +/// +public sealed class CavemanCodeCostExtractor : IAgentCostExtractor +{ + public AgentKind Kind => AgentKind.CavemanCode; + + public ModelRateConfig? DefaultPricing { get; } = null; + + private const int MaxModelIdLength = 128; + + public AgentCostSnapshot? TryExtract(string? agentStdout, string? agentStderr) + { + if (string.IsNullOrWhiteSpace(agentStdout) && string.IsNullOrWhiteSpace(agentStderr)) + return null; + + var fromStdout = TryParseJsonLines(agentStdout); + var fromStderr = TryParseJsonLines(agentStderr); + if (fromStdout is null) return fromStderr; + if (fromStderr is null) return fromStdout; + return new AgentCostSnapshot( + SaturatingAdd(fromStdout.InputTokens, fromStderr.InputTokens), + SaturatingAdd(fromStdout.CachedInputTokens, fromStderr.CachedInputTokens), + SaturatingAdd(fromStdout.OutputTokens, fromStderr.OutputTokens), + fromStdout.ModelId ?? fromStderr.ModelId); + } + + private static AgentCostSnapshot? TryParseJsonLines(string? text) + { + if (string.IsNullOrWhiteSpace(text)) return null; + // Fast reject: without a usage key no line can carry counts, and + // full JSON parsing of multi-MB prose is wasted work. + if (text.IndexOf("usage", StringComparison.OrdinalIgnoreCase) < 0 + && text.IndexOf("inputTokens", StringComparison.Ordinal) < 0) + return null; + + long input = 0, cached = 0, output = 0; + var sawUsage = false; + string? modelId = null; + + foreach (var line in text.Split('\n')) + { + var trimmed = line.Trim(); + if (trimmed.Length == 0 || trimmed[0] != '{') continue; + JsonDocument doc; + try + { + doc = JsonDocument.Parse(trimmed); + } + catch (JsonException) + { + continue; + } + + using (doc) + { + var root = doc.RootElement; + if (root.ValueKind != JsonValueKind.Object) continue; + + var usage = FindUsageObject(root); + if (usage is null) continue; + + if (TryExtractCounts(usage.Value, out var lineInput, out var lineCached, out var lineOutput) + && (lineInput > 0 || lineCached > 0 || lineOutput > 0)) + { + input = Math.Min((long)int.MaxValue, input + lineInput); + cached = Math.Min((long)int.MaxValue, cached + lineCached); + output = Math.Min((long)int.MaxValue, output + lineOutput); + sawUsage = true; + } + + modelId ??= ReadModelId(root); + } + } + + if (!sawUsage) return null; + return new AgentCostSnapshot((int)input, (int)cached, (int)output, modelId); + } + + private static JsonElement? FindUsageObject(JsonElement root) + { + if (root.TryGetProperty("usage", out var direct) + && direct.ValueKind == JsonValueKind.Object) + return direct; + + // message_end-style envelope: { ..., message: { role, usage: {...} } }. + if (root.TryGetProperty("message", out var message) + && message.ValueKind == JsonValueKind.Object + && message.TryGetProperty("usage", out var nested) + && nested.ValueKind == JsonValueKind.Object) + return nested; + + return null; + } + + private static bool TryExtractCounts(JsonElement usage, out int input, out int cached, out int output) + { + input = 0; + cached = 0; + output = 0; + + // caveman camelCase shape (field names read by the CLI's own exec + // event translator). Only the verified inputTokens/outputTokens are + // read here; no cached camelCase field has been observed, so none is + // guessed. + if (usage.TryGetProperty("inputTokens", out _) + || usage.TryGetProperty("outputTokens", out _)) + { + input = ReadNonNegative(usage, "inputTokens"); + output = ReadNonNegative(usage, "outputTokens"); + return true; + } + + // Anthropic snake_case shape (shared parser: folds cache-creation + // into fresh input, keeps cache-read separate). + if (usage.TryGetProperty("input_tokens", out _) + || usage.TryGetProperty("output_tokens", out _)) + { + AnthropicUsageParsing.ExtractUsageCounts(usage, out input, out output, out cached); + return true; + } + + // OpenAI shape. + if (usage.TryGetProperty("prompt_tokens", out _) + || usage.TryGetProperty("completion_tokens", out _)) + { + var totalInput = ReadNonNegative(usage, "prompt_tokens"); + output = ReadNonNegative(usage, "completion_tokens"); + cached = 0; + if (usage.TryGetProperty("prompt_tokens_details", out var details) + && details.ValueKind == JsonValueKind.Object) + { + cached = ReadNonNegative(details, "cached_tokens"); + } + + input = TokenUsageAccounting.FreshInputTokens(totalInput, cached); + return true; + } + + return false; + } + + private static int ReadNonNegative(JsonElement obj, string propertyName) + => obj.TryGetProperty(propertyName, out var el) + && el.ValueKind == JsonValueKind.Number + && el.TryGetInt32(out var v) + && v > 0 + ? v + : 0; + + private static string? ReadModelId(JsonElement root) + { + if (root.TryGetProperty("model", out var model) + && model.ValueKind == JsonValueKind.String) + { + var raw = model.GetString(); + if (!string.IsNullOrEmpty(raw)) + return raw.Length > MaxModelIdLength ? raw[..MaxModelIdLength] : raw; + } + + return null; + } + + private static int SaturatingAdd(int a, int b) + { + var sum = (long)a + b; + return sum > int.MaxValue ? int.MaxValue : (int)sum; + } +} diff --git a/src/CodeyBox.Agents.CavemanCode/CavemanCodeInVmSmokeProbe.cs b/src/CodeyBox.Agents.CavemanCode/CavemanCodeInVmSmokeProbe.cs new file mode 100644 index 000000000..b39514df1 --- /dev/null +++ b/src/CodeyBox.Agents.CavemanCode/CavemanCodeInVmSmokeProbe.cs @@ -0,0 +1,62 @@ +using CodeyBox.Core; + +namespace CodeyBox.Agents.CavemanCode; + +/// +/// In-VM smoke check for the caveman-code CLI: +/// +/// caveman-code --version — binary present on PATH (exit 127 otherwise). +/// When the credential bundle carries a provider API key, +/// caveman-code --list-models must exit 0 with a parseable model +/// table — proving the key reaches the CLI's environment and the model +/// registry resolves. Verified against 0.65.2: the registry answers +/// offline from env keys alone, but prints +/// No models available. Set API keys in environment variables. +/// (exit 0) when no key is visible, so the step asserts on parsed ids, +/// not just the exit code. See for +/// the shared table parser. +/// +/// +/// When no provider key is configured the probe returns only the +/// binary-presence step (still exec'd by the prober), so a binary missing +/// from PATH is caught without a false auth-failure exclusion. See +/// . +/// +public sealed class CavemanCodeInVmSmokeProbe : IInVmSmokeProbe +{ + public AgentKind Kind => AgentKind.CavemanCode; + + public IReadOnlyList BuildSteps(AgentCredential? credential) + { + var steps = new List + { + new( + [CavemanCodeAgentRunner.DefaultBinary, "--version"], + FailureHint: "caveman-code binary not runnable on sandbox PATH"), + }; + + if (HasProviderApiKey(credential)) + { + steps.Add(new( + [CavemanCodeAgentRunner.DefaultBinary, "--list-models"], + FailureHint: "caveman-code --list-models failed (no provider API key visible in-VM or registry drift)")); + } + + return steps; + } + + private static bool HasProviderApiKey(AgentCredential? credential) + { + if (credential is null) return false; + foreach (var variable in CavemanCodeAgentRunner.CredentialEnvironmentVariables) + { + if (credential.EnvironmentVariables.TryGetValue(variable, out var value) + && !string.IsNullOrEmpty(value)) + { + return true; + } + } + + return false; + } +} diff --git a/src/CodeyBox.Agents.CavemanCode/CavemanCodeModelListProbe.cs b/src/CodeyBox.Agents.CavemanCode/CavemanCodeModelListProbe.cs new file mode 100644 index 000000000..9ff596a46 --- /dev/null +++ b/src/CodeyBox.Agents.CavemanCode/CavemanCodeModelListProbe.cs @@ -0,0 +1,149 @@ +using System.Text.RegularExpressions; +using CodeyBox.Core; +using Microsoft.Extensions.Logging; + +namespace CodeyBox.Agents.CavemanCode; + +/// +/// Fetches caveman-code model identifiers by running +/// caveman-code --list-models on the host and parsing the provider +/// table from stdout. +/// +/// The table shape (provider model context max-out thinking +/// images, columns joined with two spaces) and the +/// No models available. Set API keys in environment variables. +/// empty-state line are verified against 0.65.2. The registry answers +/// offline from environment keys alone, so the probe needs the operator to +/// have provider keys (or the CODEYBOX_CAVEMAN_* host vars) visible to +/// the API host; otherwise it fails gracefully and +/// AgentClassConfigValidator skips validation with a warning. +/// +public sealed partial class CavemanCodeModelListProbe : IAgentModelListProbe +{ + internal const int MaxModelIds = 1024; + private const int MaxLoggedStderrChars = 500; + + private readonly ICavemanCodeCliRunner _runner; + private readonly string _binary; + private readonly ILogger? _log; + + public AgentKind Kind => AgentKind.CavemanCode; + + public CavemanCodeModelListProbe( + ICavemanCodeCliRunner runner, + string? binary = null, + ILogger? log = null) + { + _runner = runner; + _binary = binary ?? ResolveBinary(); + _log = log; + } + + private static string ResolveBinary() => + Environment.GetEnvironmentVariable("CODEYBOX_CAVEMANCODE_BINARY") ?? "caveman-code"; + + public async Task GetModelListAsync(CancellationToken ct) + { + try + { + var run = await _runner.RunListModelsAsync(_binary, ct).ConfigureAwait(false); + if (run.ExitCode == 1 && string.IsNullOrEmpty(run.Stdout) && string.IsNullOrEmpty(run.Stderr)) + return AgentModelListResult.Failed("caveman-code CLI failed to start"); + if (run.ExitCode == 127) + return AgentModelListResult.Failed("caveman-code CLI not found"); + if (run.ExitCode != 0) + { + LogStderrAtDebug(run.Stderr, run.ExitCode); + return AgentModelListResult.Failed($"caveman-code --list-models exited {run.ExitCode}"); + } + + var ids = ParseModelsOutput(run.Stdout); + if (ids.Count == 0) + { + LogStderrAtDebug(run.Stderr, exitCode: 0); + _log?.LogDebug("caveman-code --list-models produced no parseable model ids"); + return AgentModelListResult.Failed("no models parsed from caveman-code --list-models output"); + } + + _log?.LogDebug("caveman-code --list-models listed {Count} model id(s)", ids.Count); + return AgentModelListResult.Success(ids); + } + catch (OperationCanceledException) when (ct.IsCancellationRequested) + { + return AgentModelListResult.Failed("timeout"); + } + catch (Exception ex) when (ex is FileNotFoundException + || (ex is System.ComponentModel.Win32Exception w32 && w32.NativeErrorCode == 2)) + { + return AgentModelListResult.Failed("caveman-code CLI not found"); + } + catch (Exception ex) + { + _log?.LogDebug(ex, "caveman-code --list-models probe failed"); + return AgentModelListResult.Failed($"caveman-code --list-models failed ({ex.GetType().Name})"); + } + } + + private void LogStderrAtDebug(string stderr, int exitCode) + { + if (string.IsNullOrWhiteSpace(stderr)) return; + var trimmed = stderr.Trim(); + var capped = trimmed.Length > MaxLoggedStderrChars + ? trimmed[..MaxLoggedStderrChars] + "…" + : trimmed; + _log?.LogDebug( + "caveman-code --list-models stderr at exit {ExitCode} (len {Len}): {Stderr}", + exitCode, trimmed.Length, capped); + } + + /// + /// Parses provider/model and bare model ids from the CLI + /// table. Columns are joined with (at least) two spaces, so rows are + /// split on multi-space runs: single-spaced log noise never yields two + /// columns. The header row and the No models available… + /// empty-state line match no provider charset and are skipped. Each row + /// contributes the provider/model form first (the documented + /// --model form needing no --provider) and the bare model + /// id second, so both operator spellings validate. + /// + internal static IReadOnlyList ParseModelsOutput(string stdout, int maxIds = MaxModelIds) + { + var ids = new List(Math.Min(32, maxIds)); + var seen = new HashSet(StringComparer.OrdinalIgnoreCase); + foreach (var raw in stdout.Split('\n')) + { + var line = raw.TrimEnd(); + if (line.Length == 0) continue; + var columns = MultiSpaceRegex().Split(line.Trim()); + if (columns.Length < 2) continue; + var provider = columns[0]; + var model = columns[1]; + // Header row ("provider model context …") matches the token + // charsets below, so it is excluded by name, not by shape. + if (provider.Equals("provider", StringComparison.OrdinalIgnoreCase) + && model.Equals("model", StringComparison.OrdinalIgnoreCase)) + continue; + if (!ProviderTokenRegex().IsMatch(provider)) continue; + if (!ModelTokenRegex().IsMatch(model)) continue; + foreach (var id in new[] { $"{provider}/{model}", model }) + { + if (seen.Add(id)) + { + ids.Add(id); + if (ids.Count >= maxIds) return ids; + } + } + } + + return ids; + } + + [GeneratedRegex(@"\s{2,}")] + private static partial Regex MultiSpaceRegex(); + + [GeneratedRegex("^[a-z0-9_.-]+$", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex ProviderTokenRegex(); + + [GeneratedRegex("^[a-z0-9_.\\-/:+]+$", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex ModelTokenRegex(); +} diff --git a/src/CodeyBox.Agents.CavemanCode/CavemanCodeQuotaFailureDetector.cs b/src/CodeyBox.Agents.CavemanCode/CavemanCodeQuotaFailureDetector.cs new file mode 100644 index 000000000..9b873ce78 --- /dev/null +++ b/src/CodeyBox.Agents.CavemanCode/CavemanCodeQuotaFailureDetector.cs @@ -0,0 +1,81 @@ +using CodeyBox.Agents; +using CodeyBox.Core; + +namespace CodeyBox.Agents.CavemanCode; + +/// +/// Recognises auth / rate-limit failures emitted by the caveman-code CLI. +/// +/// Every pattern below is verified, not guessed: +/// +/// The two Unauthorized rows are the CLI's own missing-key +/// output, observed live against 0.65.2 +/// (No API key found for unknown. followed by +/// Use /login or set an API key environment variable.). +/// overloaded_error is the provider error code the CLI's own +/// retry classifier matches (shipped dist/core/agent-session.js, +/// 0.65.2); anchored with the _error suffix so prose about an +/// "overloaded" server under review does not gate dispatch. +/// Numeric 429 rows come from +/// and +/// stay anchored with companion text for the same reason — a bare +/// 429 in code under review must not bench the agent. +/// +/// +/// Hard-quota shapes (provider spend caps) are deliberately absent: +/// BYOK keys have no single quota meter and no such stderr shape has been +/// observed. They get added reactively once a real failure is seen in +/// production, mirroring the opencode detector's stance. There is no +/// IAgentQuotaProbe for caveman-code for the same reason. +/// +public sealed class CavemanCodeQuotaFailureDetector : IAgentQuotaFailureDetector +{ + public AgentKind Kind => AgentKind.CavemanCode; + + private static readonly QuotaFailurePattern[] Patterns = + [ + // Transient provider refusals relayed verbatim by the CLI — owned by + // SharedRateLimitPatterns so caveman-code stays in step with the + // Copilot/opencode detectors. Checked first so a refusal carrying + // both shapes parks on the rate-limit backoff. + .. Agents.SharedRateLimitPatterns.ProviderRateLimitPatterns, + // Provider-side retryable error code matched by the CLI's own retry + // classifier (agent-session.js, 0.65.2). Suffix-anchored: bare + // "overloaded" prose must not trip a false positive. + new("overloaded_error", QuotaFailureKind.RateLimitExceeded), + // Missing-key output observed live (0.65.2). Both lines are matched + // so a truncated capture still classifies. + new("No API key found", QuotaFailureKind.Unauthorized), + new("set an API key environment variable", QuotaFailureKind.Unauthorized), + // Anchor with the HTTP status so the bare word "Unauthorized" in + // model output (e.g. discussing access-control code) doesn't trigger. + new("401 Unauthorized", QuotaFailureKind.Unauthorized), + new("API Error: 401", QuotaFailureKind.Unauthorized), + ]; + + public QuotaDetection? Detect(string? stderr, string? stdout) + { + if (string.IsNullOrEmpty(stderr) && string.IsNullOrEmpty(stdout)) + return null; + + foreach (var (pattern, kind) in Patterns) + { + var inStderr = !string.IsNullOrEmpty(stderr) + && stderr.Contains(pattern, StringComparison.OrdinalIgnoreCase); + var inStdout = !string.IsNullOrEmpty(stdout) + && stdout.Contains(pattern, StringComparison.OrdinalIgnoreCase); + if (inStderr || inStdout) + { + var sources = new List(2); + if (!string.IsNullOrEmpty(stderr)) sources.Add(stderr); + if (!string.IsNullOrEmpty(stdout)) sources.Add(stdout); + return new QuotaDetection( + kind, + QuotaResetParser.TryParseResetAt(sources) + ?? QuotaResetParser.TryParseRetryAfterHeader(sources)); + } + } + + return null; + } +} diff --git a/src/CodeyBox.Agents.CavemanCode/CavemanCodeSmokeProbe.cs b/src/CodeyBox.Agents.CavemanCode/CavemanCodeSmokeProbe.cs new file mode 100644 index 000000000..ad97dd0c9 --- /dev/null +++ b/src/CodeyBox.Agents.CavemanCode/CavemanCodeSmokeProbe.cs @@ -0,0 +1,47 @@ +using CodeyBox.Core; +using Microsoft.Extensions.Logging; + +namespace CodeyBox.Agents.CavemanCode; + +/// +/// Minimal credential viability check for caveman-code. Returns Ok when the +/// bundle contains a non-empty provider API key from +/// +/// (BYOK auth, shipped to the sandbox environment by the runner); +/// returns Fail otherwise. +/// +/// This does NOT issue a network call: with no key configured the CLI +/// itself reports No API key found (verified live against 0.65.2), +/// and with a key the static model registry answers --list-models +/// offline, so a host-side HTTP probe would add no signal. Key validity is +/// verified in-VM by (the table must +/// parse) and at dispatch by the quota-failure detector. Mirrors the +/// opencode credential-presence probe shape. +/// +public sealed class CavemanCodeSmokeProbe : IAgentSmokeProbe +{ + private readonly ILogger? _log; + + public AgentKind Kind => AgentKind.CavemanCode; + + public CavemanCodeSmokeProbe(ILogger? log = null) + { + _log = log; + } + + public Task SmokeTestAsync(AgentCredential credential, CancellationToken ct) + { + foreach (var variable in CavemanCodeAgentRunner.CredentialEnvironmentVariables) + { + if (credential.EnvironmentVariables.TryGetValue(variable, out var value) + && !string.IsNullOrEmpty(value)) + { + return Task.FromResult(new AgentSmokeResult(true, null, TimeSpan.Zero, SmokeFailureCategory.None)); + } + } + + _log?.LogDebug("CavemanCode smoke probe found no provider API key in credential bundle"); + return Task.FromResult(new AgentSmokeResult( + false, "no provider API key in credential bundle", TimeSpan.Zero, SmokeFailureCategory.Persistent)); + } +} diff --git a/src/CodeyBox.Agents.CavemanCode/CavemanCodeStreamParser.cs b/src/CodeyBox.Agents.CavemanCode/CavemanCodeStreamParser.cs new file mode 100644 index 000000000..1b713898a --- /dev/null +++ b/src/CodeyBox.Agents.CavemanCode/CavemanCodeStreamParser.cs @@ -0,0 +1,35 @@ +using System.Text.Json; +using CodeyBox.Agents; +using CodeyBox.Core; + +namespace CodeyBox.Agents.CavemanCode; + +/// +/// Stream parser slot for caveman-code. The runner invokes +/// caveman-code -p in plain-text mode (print-mode --mode json +/// emits the CLI's internal unfrozen session events, and the frozen +/// exec --json stream cannot take the prompt on stdin — see +/// ), so the captured stream file is +/// plaintext stdout/stderr. This parser exists so that +/// resolves caveman-code +/// work items to rather than +/// unknown; the inherited +/// returns +/// for plaintext output, at +/// which point re-runs the file through +/// the plaintext-fallback summariser. +/// +/// If a future release documents a stable stream shape (today only the +/// exec --json event envelope is frozen, and the runner does not use +/// it), override to recognise it. Until then this +/// parser claims nothing by shape. +/// +public sealed class CavemanCodeStreamParser : FlexibleAgentStreamParser +{ + public CavemanCodeStreamParser(AgentStreamParserOptions? options = null) + : base(AgentKind.CavemanCode, options) + { + } + + public override bool TryClaim(JsonElement line) => false; +} diff --git a/src/CodeyBox.Agents.CavemanCode/CodeyBox.Agents.CavemanCode.csproj b/src/CodeyBox.Agents.CavemanCode/CodeyBox.Agents.CavemanCode.csproj new file mode 100644 index 000000000..ff8aae0c7 --- /dev/null +++ b/src/CodeyBox.Agents.CavemanCode/CodeyBox.Agents.CavemanCode.csproj @@ -0,0 +1,19 @@ + + + + + + + + + + net10.0 + enable + enable + + + + + + + diff --git a/src/CodeyBox.Agents.CavemanCode/DefaultCavemanCodeCliRunner.cs b/src/CodeyBox.Agents.CavemanCode/DefaultCavemanCodeCliRunner.cs new file mode 100644 index 000000000..ba7ea9093 --- /dev/null +++ b/src/CodeyBox.Agents.CavemanCode/DefaultCavemanCodeCliRunner.cs @@ -0,0 +1,38 @@ +using CodeyBox.HostProcess; + +namespace CodeyBox.Agents.CavemanCode; + +/// +/// Host process runner for caveman-code --list-models via shared +/// . +/// +public sealed class DefaultCavemanCodeCliRunner : ICavemanCodeCliRunner +{ + private const int MaxOutputBytes = 512 * 1024; + private readonly IProcessRunner _runner; + private readonly IReadOnlyDictionary? _environment; + + public DefaultCavemanCodeCliRunner( + IProcessRunner? runner = null, + IReadOnlyDictionary? environment = null) + { + _runner = runner ?? new DefaultProcessRunner(); + _environment = environment ?? MinimalHostProcessEnvironment.ForCliAuthDiscovery(); + } + + public async Task RunListModelsAsync(string binary, CancellationToken ct) + { + var result = await _runner.RunAsync( + [binary, "--list-models"], + stdin: null, + ct, + maxStdoutBytes: MaxOutputBytes, + maxStderrBytes: MaxOutputBytes, + environment: _environment).ConfigureAwait(false); + + if (result.StartFailed) + return new CavemanCodeCliRunResult(1, "", ""); + + return new CavemanCodeCliRunResult(result.ExitCode, result.Stdout, result.Stderr); + } +} diff --git a/src/CodeyBox.Agents.CavemanCode/ICavemanCodeCliRunner.cs b/src/CodeyBox.Agents.CavemanCode/ICavemanCodeCliRunner.cs new file mode 100644 index 000000000..12a214978 --- /dev/null +++ b/src/CodeyBox.Agents.CavemanCode/ICavemanCodeCliRunner.cs @@ -0,0 +1,18 @@ +namespace CodeyBox.Agents.CavemanCode; + +/// +/// Runs the caveman-code CLI on the host for probes that need local output. +/// Abstracted so unit tests and the API composition root can substitute process +/// execution without leaking . +/// +public interface ICavemanCodeCliRunner +{ + /// + /// Runs {binary} --list-models and returns exit code plus captured streams. + /// + /// When cannot be executed. + /// On Linux/macOS when the binary is absent from PATH (ENOENT). + Task RunListModelsAsync(string binary, CancellationToken ct); +} + +public readonly record struct CavemanCodeCliRunResult(int ExitCode, string Stdout, string Stderr); diff --git a/src/CodeyBox.Api/CodeyBox.Api.csproj b/src/CodeyBox.Api/CodeyBox.Api.csproj index 40184f228..03332b9cd 100644 --- a/src/CodeyBox.Api/CodeyBox.Api.csproj +++ b/src/CodeyBox.Api/CodeyBox.Api.csproj @@ -16,6 +16,7 @@ + diff --git a/src/CodeyBox.Api/Program.cs b/src/CodeyBox.Api/Program.cs index 1b988fd43..f827a4794 100644 --- a/src/CodeyBox.Api/Program.cs +++ b/src/CodeyBox.Api/Program.cs @@ -7,6 +7,7 @@ using OpenTelemetry.Trace; using CodeyBox.Agents; using CodeyBox.Agents.Antigravity; +using CodeyBox.Agents.CavemanCode; using CodeyBox.Agents.Crock; using CodeyBox.Agents.Claude; using CodeyBox.Agents.Codex; @@ -1156,6 +1157,8 @@ static string FormatBytes(long bytes) sp.GetRequiredService())); builder.Services.AddSingleton(); builder.Services.AddSingleton(); +builder.Services.AddSingleton(sp => new CavemanCodeAgentRunner( + sp.GetRequiredService())); builder.Services.AddSingleton(sp => new AntigravityAgentRunner { // agy's built-in --print-timeout default (5m) aborts a one-shot session with @@ -1582,6 +1585,14 @@ static string FormatBytes(long bytes) CopilotAgentRunner.ProviderBearerTokenEnvironmentVariable), new AgentCredentialMapping(AgentKind.Codex, "CODEYBOX_CODEX_API_KEY", "OPENAI_API_KEY"), new AgentCredentialMapping(AgentKind.Gemini, "CODEYBOX_GEMINI_API_KEY", "GEMINI_API_KEY"), + // CavemanCode is BYOK across providers: the CLI reads whichever + // provider key is present (ANTHROPIC_API_KEY, OPENAI_API_KEY, … — + // see CavemanCodeAgentRunner.CredentialEnvironmentVariables). The + // namespaced host vars keep secrets out of config files. + new AgentCredentialMapping(AgentKind.CavemanCode, "CODEYBOX_CAVEMAN_ANTHROPIC_API_KEY", "ANTHROPIC_API_KEY"), + new AgentCredentialMapping(AgentKind.CavemanCode, "CODEYBOX_CAVEMAN_OPENAI_API_KEY", "OPENAI_API_KEY"), + new AgentCredentialMapping(AgentKind.CavemanCode, "CODEYBOX_CAVEMAN_GEMINI_API_KEY", "GEMINI_API_KEY"), + new AgentCredentialMapping(AgentKind.CavemanCode, "CODEYBOX_CAVEMAN_OPENROUTER_API_KEY", "OPENROUTER_API_KEY"), // Cursor: the CLI uses subscription auth via ~/.cursor/credentials.json // (NOT an env-var key). The orchestrator ships the file's contents to // the sandbox via CODEYBOX_CURSOR_AUTH_JSON and CursorAgentRunner @@ -1631,6 +1642,16 @@ static string FormatBytes(long bytes) // audit runners authenticated in hosts that inject OPENAI_API_KEY // directly instead of the CodeyBox-namespaced variant above. new AgentCredentialMapping(AgentKind.Codex, "OPENAI_API_KEY", "OPENAI_API_KEY"), + // CavemanCode verbatim fallbacks live in this separate provider (not + // alongside the CODEYBOX_CAVEMAN_* rows above): one provider instance + // rejects two mappings for the same agent targeting the same sandbox + // variable, so the Codex precedent keeps namespaced and verbatim rows + // in different instances. These let hosts that already inject the + // conventional provider keys work without extra operator wiring. + new AgentCredentialMapping(AgentKind.CavemanCode, "ANTHROPIC_API_KEY", "ANTHROPIC_API_KEY"), + new AgentCredentialMapping(AgentKind.CavemanCode, "OPENAI_API_KEY", "OPENAI_API_KEY"), + new AgentCredentialMapping(AgentKind.CavemanCode, "GEMINI_API_KEY", "GEMINI_API_KEY"), + new AgentCredentialMapping(AgentKind.CavemanCode, "OPENROUTER_API_KEY", "OPENROUTER_API_KEY"), })); return new ChainedCredentialProvider( @@ -2286,6 +2307,9 @@ static IAgentQuotaProbe WireQuotaProbeTokenInvalidation( builder.Services.AddSingleton(sp => new OpencodeSmokeProbe( sp.GetRequiredService().CreateLogger())); +builder.Services.AddSingleton(sp => + new CavemanCodeSmokeProbe( + sp.GetRequiredService().CreateLogger())); builder.Services.AddSingleton(sp => new AntigravitySmokeProbe( sp.GetRequiredService(), @@ -2308,6 +2332,7 @@ static IAgentQuotaProbe WireQuotaProbeTokenInvalidation( builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); +builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); @@ -2363,6 +2388,14 @@ static IAgentQuotaProbe WireQuotaProbeTokenInvalidation( loggerFactory.CreateLogger()); }); builder.Services.AddSingleton(sp => +{ + var loggerFactory = sp.GetRequiredService(); + return new CavemanCodeModelListProbe( + new DefaultCavemanCodeCliRunner(), + binary: Environment.GetEnvironmentVariable("CODEYBOX_CAVEMANCODE_BINARY"), + loggerFactory.CreateLogger()); +}); +builder.Services.AddSingleton(sp => { var loggerFactory = sp.GetRequiredService(); var source = sp.GetRequiredService(); @@ -3407,6 +3440,7 @@ static Func DotnetTestRunOptionsAccessor(IServiceProvider sp) [AgentKind.Antigravity] = new AntigravityCostExtractor(), [AgentKind.Crock] = new CrockCostExtractor(), [AgentKind.Pi] = new PiCostExtractor(), + [AgentKind.CavemanCode] = new CavemanCodeCostExtractor(), }; // Warn once at startup for registered agents with no extractor. foreach (var kind in registry.Available) @@ -3499,6 +3533,7 @@ static Func DotnetTestRunOptionsAccessor(IServiceProvider sp) builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); +builder.Services.AddSingleton(); builder.Services.AddSingleton(); // Per-provider buffered-stdout tool-call counters. Used by the orchestrator @@ -3549,6 +3584,7 @@ static Func DotnetTestRunOptionsAccessor(IServiceProvider sp) .ToArray(); return new PiQuotaFailureDetector(extras); }); +builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(sp => diff --git a/src/CodeyBox.Api/WorkItemDiffEndpoints.cs b/src/CodeyBox.Api/WorkItemDiffEndpoints.cs index 9515a6078..79641fa6e 100644 --- a/src/CodeyBox.Api/WorkItemDiffEndpoints.cs +++ b/src/CodeyBox.Api/WorkItemDiffEndpoints.cs @@ -264,6 +264,11 @@ private static ProcessStartInfo BuildGitDiffPsi(string repoPath, string baseSha, UseShellExecute = false, CreateNoWindow = true, }; + // --git-dir is explicit (not cwd discovery): hardened hosts that set + // safe.bareRepository=explicit refuse to treat a bare repo found via + // the working directory as a repository, so plain `git diff` fails + // there with "Not a git repository". An explicit git dir is honored. + psi.ArgumentList.Add("--git-dir=" + repoPath); psi.ArgumentList.Add("diff"); psi.ArgumentList.Add("--unified=3"); psi.ArgumentList.Add($"{baseSha}..{workSha}"); @@ -282,6 +287,9 @@ private static ProcessStartInfo BuildGitDiffPsi(string repoPath, string baseSha, UseShellExecute = false, CreateNoWindow = true, }; + // See BuildGitDiffPsi: explicit --git-dir keeps bare-repo commands + // working under safe.bareRepository=explicit. + psi.ArgumentList.Add("--git-dir=" + workdir); foreach (var a in args) psi.ArgumentList.Add(a); using var p = System.Diagnostics.Process.Start(psi)!; var readStdout = p.StandardOutput.ReadToEndAsync(ct); diff --git a/src/CodeyBox.Api/agent-pricing-defaults.json b/src/CodeyBox.Api/agent-pricing-defaults.json index f7d2e6c82..8482c4bc0 100644 --- a/src/CodeyBox.Api/agent-pricing-defaults.json +++ b/src/CodeyBox.Api/agent-pricing-defaults.json @@ -5,6 +5,7 @@ "opencode": "https://opencode.ai/docs/go", "claude": "https://docs.anthropic.com/en/docs/about-claude/pricing", "codex": "https://openai.com/api/pricing/", + "caveman": "https://openai.com/api/pricing/ + https://docs.anthropic.com/en/docs/about-claude/pricing (same per-token rates as the codex/claude buckets; caveman-code is BYOK so spend bills to the operator's own provider accounts)", "gemini": "https://ai.google.dev/gemini-api/docs/pricing", "pi": "https://docs.anthropic.com/en/docs/about-claude/pricing" }, @@ -12,6 +13,7 @@ "opencode": "OpenCode Go is subscription-priced ($12 per 5h usage budget on https://opencode.ai/docs/go). Each model entry uses one subscription-equivalent USD/M rate for input, cached input, and output: ($12 ÷ requests-per-5h) ÷ ((typical input + cached + output tokens per request) ÷ 1e6), using the request limits and typical token mix published on that page. Keys use opencode-go/. Cost attribution charges the stored non-cached input bucket and cached input bucket separately; public total-input reporting adds them.", "claude": "Cache-hit (read) rate captured under cachedInputPerMillion. Cache-write rates (1.25x base for 5m, 2x for 1h) are not represented; cost framework treats CachedInputTokens as read-side discount.", "codex": "gpt-5.5 short-context rate ($5/$30). codex-5.5 aliases the same rate for CLI JSON attribution. Long-context tier (above ~270k input tokens) is not represented.", + "caveman": "Caveman-code is BYOK: the operator pays the backing provider directly, so rates mirror the codex bucket (OpenAI ids) and the claude bucket (Anthropic ids) for the same model ids. Keys ship in both bare (gpt-5.5) and provider-prefixed (openai/gpt-5.5) forms because the CLI reports either depending on how --model was given. Only registry-verified ids are seeded; add further provider ids under CodeyBox:AgentPricing as operators adopt them.", "gemini": "gemini-3-flash-preview is the default AgentClasses model. Pro preview models use short-context rates (<= 200k tokens); long-context tier (>200k) is not represented.", "cursor": "Cursor uses a flat subscription and does not publish per-token rates; intentionally omitted from defaults.", "copilot": "Copilot uses a flat subscription and does not publish per-token rates; intentionally omitted from defaults.", @@ -53,6 +55,16 @@ "codex-5.5": { "inputPerMillion": 5.00, "cachedInputPerMillion": 0.50, "outputPerMillion": 30.00 }, "gpt-5.5-pro": { "inputPerMillion": 30.00, "cachedInputPerMillion": 3.00, "outputPerMillion": 180.00 } }, + "caveman": { + "gpt-5.5": { "inputPerMillion": 5.00, "cachedInputPerMillion": 0.50, "outputPerMillion": 30.00 }, + "openai/gpt-5.5": { "inputPerMillion": 5.00, "cachedInputPerMillion": 0.50, "outputPerMillion": 30.00 }, + "claude-opus-4-6": { "inputPerMillion": 5.00, "cachedInputPerMillion": 0.50, "outputPerMillion": 25.00 }, + "anthropic/claude-opus-4-6": { "inputPerMillion": 5.00, "cachedInputPerMillion": 0.50, "outputPerMillion": 25.00 }, + "claude-opus-4-7": { "inputPerMillion": 5.00, "cachedInputPerMillion": 0.50, "outputPerMillion": 25.00 }, + "anthropic/claude-opus-4-7": { "inputPerMillion": 5.00, "cachedInputPerMillion": 0.50, "outputPerMillion": 25.00 }, + "claude-sonnet-4-6": { "inputPerMillion": 3.00, "cachedInputPerMillion": 0.30, "outputPerMillion": 15.00 }, + "anthropic/claude-sonnet-4-6": { "inputPerMillion": 3.00, "cachedInputPerMillion": 0.30, "outputPerMillion": 15.00 } + }, "gemini": { "gemini-3-flash-preview": { "inputPerMillion": 0.50, "cachedInputPerMillion": 0.05, "outputPerMillion": 3.00 }, "gemini-3-pro-preview": { "inputPerMillion": 2.00, "cachedInputPerMillion": 0.20, "outputPerMillion": 12.00 }, diff --git a/src/CodeyBox.Api/appsettings.json b/src/CodeyBox.Api/appsettings.json index 64b4b74cc..0e4418a0c 100644 --- a/src/CodeyBox.Api/appsettings.json +++ b/src/CodeyBox.Api/appsettings.json @@ -95,7 +95,8 @@ "claude": "claude-opus-4-7", "codex": "gpt-5.5", "cursor": "composer-2.5", - "opencode": "deepseek-v4-flash" + "opencode": "deepseek-v4-flash", + "caveman": "openai/gpt-5.5" }, "AgentNetworkTolerance": { "codex": { @@ -138,7 +139,8 @@ { "Agent": "codex", "Billing": "Subscription", "ModelId": "gpt-5.5", "QualityScore": 100, "Capabilities": [ "sensitive" ] }, { "Agent": "cursor", "Billing": "Subscription", "ModelId": "composer-2.5", "QualityScore": 98 }, { "Agent": "gemini", "Billing": "Subscription", "ModelId": "gemini-3-flash-preview", "QualityScore": 95, "ReasoningMode": "high" }, - { "Agent": "pi", "Billing": "PayPerApi", "ModelId": "anthropic/claude-haiku-4-5", "QualityScore": 90 } + { "Agent": "pi", "Billing": "PayPerApi", "ModelId": "anthropic/claude-haiku-4-5", "QualityScore": 90 }, + { "Agent": "caveman", "Billing": "PayPerApi", "ModelId": "openai/gpt-5.5", "QualityScore": 95 } ] } ], diff --git a/src/CodeyBox.Core/AgentKind.cs b/src/CodeyBox.Core/AgentKind.cs index d9bdafc03..5851635fa 100644 --- a/src/CodeyBox.Core/AgentKind.cs +++ b/src/CodeyBox.Core/AgentKind.cs @@ -15,6 +15,7 @@ public readonly record struct AgentKind(string Value) public static AgentKind Antigravity { get; } = new("antigravity"); public static AgentKind Crock { get; } = new("crock"); public static AgentKind Pi { get; } = new("pi"); + public static AgentKind CavemanCode { get; } = new("caveman"); public override string ToString() => Value; } diff --git a/tests/CodeyBox.Tests/CavemanCodeAgentRunnerTests.cs b/tests/CodeyBox.Tests/CavemanCodeAgentRunnerTests.cs new file mode 100644 index 000000000..20a7ef63e --- /dev/null +++ b/tests/CodeyBox.Tests/CavemanCodeAgentRunnerTests.cs @@ -0,0 +1,147 @@ +using CodeyBox.Agents.CavemanCode; +using CodeyBox.Core; + +namespace CodeyBox.Tests; + +/// +/// Unit tests for . Uses the shared +/// CapturingSandbox to inspect the argv and stdin the runner forwards. +/// +public sealed class CavemanCodeAgentRunnerTests +{ + [Fact] + public void Kind_IsCavemanCode() + { + Assert.Equal(AgentKind.CavemanCode, new CavemanCodeAgentRunner().Kind); + } + + [Fact] + public void AgentKind_CavemanCode_RoundTrips() + { + Assert.Equal(AgentKind.CavemanCode, new AgentKind("caveman")); + } + + [Fact] + public async Task RunAsync_Argv_UsesCavemanCodeBinaryWithPrintFlag() + { + var sandbox = new CapturingSandbox(); + var runner = new CavemanCodeAgentRunner(); + + await runner.RunAsync(sandbox, "/work", "do the thing", credential: null); + + var argv = sandbox.CapturedExec!.Argv; + Assert.Equal("caveman-code", argv[0]); + Assert.Equal("-p", argv[1]); + } + + [Fact] + public async Task RunAsync_Prompt_GoesToStdinNotArgv() + { + // MAX_ARG_STRLEN is 128 KiB per argv element; the runner feeds the + // prompt via stdin (verified: `echo ... | caveman-code -p` reaches + // agent init with no positional message). + var sandbox = new CapturingSandbox(); + var runner = new CavemanCodeAgentRunner(); + const string prompt = "a prompt far too interesting to inline in argv"; + + await runner.RunAsync(sandbox, "/work", prompt, credential: null); + + Assert.Equal(prompt, sandbox.CapturedExec!.Stdin); + Assert.DoesNotContain(sandbox.CapturedExec.Argv, a => a.Contains(prompt)); + } + + [Fact] + public async Task RunAsync_DefaultModel_FromAgentDefaults() + { + var sandbox = new CapturingSandbox(); + var defaults = new AgentDefaultsSnapshot( + new Dictionary(StringComparer.OrdinalIgnoreCase) + { + ["caveman"] = "openai/gpt-5.5", + }); + var runner = new CavemanCodeAgentRunner(defaults); + + await runner.RunAsync(sandbox, "/work", "x", credential: null); + + var argv = sandbox.CapturedExec!.Argv; + Assert.Contains("--model", argv); + Assert.Contains("openai/gpt-5.5", argv); + } + + [Fact] + public async Task RunAsync_ExplicitModelId_OverridesDefault() + { + var sandbox = new CapturingSandbox(); + var defaults = new AgentDefaultsSnapshot( + new Dictionary(StringComparer.OrdinalIgnoreCase) + { + ["caveman"] = "openai/gpt-5.5", + }); + var runner = new CavemanCodeAgentRunner(defaults); + + await runner.RunAsync(sandbox, "/work", "x", credential: null, + modelId: "anthropic/claude-opus-4-6"); + + var argv = sandbox.CapturedExec!.Argv.ToList(); + var modelIdx = argv.IndexOf("--model"); + Assert.True(modelIdx >= 0); + Assert.Equal("anthropic/claude-opus-4-6", argv[modelIdx + 1]); + } + + [Fact] + public async Task RunAsync_NoModelAnywhere_OmitsModelFlag() + { + // Without an explicit id or a configured default the CLI resolves + // its own per-provider default — the runner must not invent one. + var sandbox = new CapturingSandbox(); + var runner = new CavemanCodeAgentRunner(); + + await runner.RunAsync(sandbox, "/work", "x", credential: null); + + Assert.DoesNotContain("--model", sandbox.CapturedExec!.Argv); + } + + [Fact] + public async Task RunAsync_ReasoningModeHigh_ForwardsThinkingFlag() + { + var sandbox = new CapturingSandbox(); + var runner = new CavemanCodeAgentRunner(); + + await runner.RunAsync(sandbox, "/work", "x", credential: null, reasoningMode: "high"); + + var argv = sandbox.CapturedExec!.Argv.ToList(); + var thinkingIdx = argv.IndexOf("--thinking"); + Assert.True(thinkingIdx >= 0); + Assert.Equal("high", argv[thinkingIdx + 1]); + } + + [Fact] + public async Task RunAsync_UnknownReasoningMode_DropsFlag() + { + // An unrecognised level must not reach the CLI: it would only add a + // startup warning while silently running at the CLI default. + var sandbox = new CapturingSandbox(); + var runner = new CavemanCodeAgentRunner(); + + await runner.RunAsync(sandbox, "/work", "x", credential: null, reasoningMode: "ultra"); + + Assert.DoesNotContain("--thinking", sandbox.CapturedExec!.Argv); + } + + [Fact] + public async Task RunAsync_NeverPassesApiKeyFlag() + { + // Secrets must not ride argv (visible in process listings); keys + // arrive via the sandbox environment as direct credential vars. + var sandbox = new CapturingSandbox(); + var runner = new CavemanCodeAgentRunner(); + var cred = new AgentCredential(AgentKind.CavemanCode, + new Dictionary { ["OPENAI_API_KEY"] = "sk-test" }, + new Dictionary()); + + await runner.RunAsync(sandbox, "/work", "x", credential: cred); + + Assert.DoesNotContain("--api-key", sandbox.CapturedExec!.Argv); + Assert.DoesNotContain(sandbox.CapturedExec.Argv, a => a.Contains("sk-test")); + } +} diff --git a/tests/CodeyBox.Tests/CavemanCodeCostExtractorTests.cs b/tests/CodeyBox.Tests/CavemanCodeCostExtractorTests.cs new file mode 100644 index 000000000..38940aab1 --- /dev/null +++ b/tests/CodeyBox.Tests/CavemanCodeCostExtractorTests.cs @@ -0,0 +1,149 @@ +using CodeyBox.Agents.CavemanCode; +using CodeyBox.Core; + +namespace CodeyBox.Tests; + +/// +/// Tests for . Only machine-shaped JSON +/// usage envelopes are parsed — prose token mentions must never fabricate +/// spend, because plain-text -p runs emit no usage footer at all. +/// +public sealed class CavemanCodeCostExtractorTests +{ + private static readonly CavemanCodeCostExtractor Extractor = new(); + + [Fact] + public void Kind_IsCavemanCode() + { + Assert.Equal(AgentKind.CavemanCode, Extractor.Kind); + } + + [Fact] + public void DefaultPricing_IsNull() + { + // caveman-code fronts many providers with different per-token + // economics; there is no sensible single fallback rate. Operators + // configure per-model pricing under CodeyBox:AgentPricing. + Assert.Null(Extractor.DefaultPricing); + } + + [Fact] + public void NullAndEmpty_ReturnsNull() + { + Assert.Null(Extractor.TryExtract(null, null)); + Assert.Null(Extractor.TryExtract("", "")); + Assert.Null(Extractor.TryExtract(" ", null)); + Assert.Null(Extractor.TryExtract(null, " ")); + } + + [Fact] + public void PlainAssistantProse_ReturnsNull() + { + // The critical negative: text-mode output carries no usage envelope, + // so even prose bragging about tokens must not produce a snapshot. + const string prose = "I used about 1500 input tokens and 250 output tokens for this change."; + Assert.Null(Extractor.TryExtract(prose, null)); + } + + [Fact] + public void Json_CavemanCamelCaseShape_ParsesInputAndOutput() + { + var stdout = """{"type":"message_end","message":{"role":"assistant","usage":{"inputTokens":1500,"outputTokens":250}}}"""; + + var result = Extractor.TryExtract(stdout, null); + + Assert.NotNull(result); + Assert.Equal(1500, result!.InputTokens); + Assert.Equal(250, result.OutputTokens); + Assert.Equal(0, result.CachedInputTokens); + } + + [Fact] + public void Json_OpenAiShape_ParsesPromptAndCompletionTokens() + { + var stdout = """{"usage":{"prompt_tokens":1500,"completion_tokens":250}}"""; + + var result = Extractor.TryExtract(stdout, null); + + Assert.NotNull(result); + Assert.Equal(1500, result!.InputTokens); + Assert.Equal(250, result.OutputTokens); + } + + [Fact] + public void Json_OpenAiCachedTokens_SplitIntoCachedBucket() + { + var stdout = """{"usage":{"prompt_tokens":1500,"completion_tokens":250,"prompt_tokens_details":{"cached_tokens":500}}}"""; + + var result = Extractor.TryExtract(stdout, null); + + Assert.NotNull(result); + Assert.Equal(1000, result!.InputTokens); + Assert.Equal(500, result.CachedInputTokens); + Assert.Equal(250, result.OutputTokens); + } + + [Fact] + public void Json_AnthropicShape_ParsesInputAndOutputTokens() + { + var stdout = """{"usage":{"input_tokens":2000,"output_tokens":300,"cache_read_input_tokens":400,"cache_creation_input_tokens":100}}"""; + + var result = Extractor.TryExtract(stdout, null); + + Assert.NotNull(result); + Assert.Equal(2100, result!.InputTokens); + Assert.Equal(400, result.CachedInputTokens); + Assert.Equal(300, result.OutputTokens); + } + + [Fact] + public void Json_ModelId_RecordedAndBounded() + { + var stdout = """{"model":"openai/gpt-5.5","usage":{"inputTokens":10,"outputTokens":5}}"""; + + var result = Extractor.TryExtract(stdout, null); + + Assert.NotNull(result); + Assert.Equal("openai/gpt-5.5", result!.ModelId); + } + + [Fact] + public void Json_MultipleUsageLines_AccumulateSessionTotal() + { + var stdout = string.Join('\n', + """{"type":"message_end","message":{"role":"assistant","usage":{"inputTokens":100,"outputTokens":10}}}""", + """{"type":"message_end","message":{"role":"assistant","usage":{"inputTokens":200,"outputTokens":20}}}"""); + + var result = Extractor.TryExtract(stdout, null); + + Assert.NotNull(result); + Assert.Equal(300, result!.InputTokens); + Assert.Equal(30, result.OutputTokens); + } + + [Fact] + public void Json_NegativeCounters_Ignored() + { + var stdout = """{"usage":{"inputTokens":-50,"outputTokens":20}}"""; + + var result = Extractor.TryExtract(stdout, null); + + Assert.NotNull(result); + Assert.Equal(0, result!.InputTokens); + Assert.Equal(20, result.OutputTokens); + } + + [Fact] + public void Json_InterleavedProse_LinesSkipped() + { + var stdout = string.Join('\n', + "Some log chatter without braces", + """{"usage":{"inputTokens":100,"outputTokens":10}}""", + "{not valid json"); + + var result = Extractor.TryExtract(stdout, null); + + Assert.NotNull(result); + Assert.Equal(100, result!.InputTokens); + } +} diff --git a/tests/CodeyBox.Tests/CavemanCodeModelListProbeTests.cs b/tests/CodeyBox.Tests/CavemanCodeModelListProbeTests.cs new file mode 100644 index 000000000..c7f03369c --- /dev/null +++ b/tests/CodeyBox.Tests/CavemanCodeModelListProbeTests.cs @@ -0,0 +1,199 @@ +using System.ComponentModel; +using CodeyBox.Agents.CavemanCode; +using CodeyBox.Core; +using CodeyBox.HostProcess; + +namespace CodeyBox.Tests; + +/// +/// Unit tests for parsing and CLI +/// outcomes. The table fixture is verbatim +/// caveman-code --list-models output captured against 0.65.2 (public +/// model catalog — no secrets to redact, suffix kept for convention). +/// +public sealed class CavemanCodeModelListProbeTests +{ + private static string FixturePath => + Path.Combine(AppContext.BaseDirectory, "Fixtures", "CavemanCode", "caveman-code-list-models.redacted.txt"); + + [Fact] + public void Kind_IsCavemanCode() + => Assert.Equal( + AgentKind.CavemanCode, + new CavemanCodeModelListProbe(new StubCavemanCodeCliRunner(0, "", "")).Kind); + + [Fact] + public void ParseModelsOutput_Fixture_YieldsPrefixedAndBareIds() + { + var stdout = File.ReadAllText(FixturePath); + var ids = CavemanCodeModelListProbe.ParseModelsOutput(stdout); + + Assert.Equal( + new[] + { + "anthropic/claude-haiku-4-5", "claude-haiku-4-5", + "anthropic/claude-opus-4-6", "claude-opus-4-6", + "anthropic/claude-sonnet-4-6", "claude-sonnet-4-6", + "openai/gpt-5-codex", "gpt-5-codex", + "openai/gpt-5.5", "gpt-5.5", + }, + ids); + } + + [Fact] + public void ParseModelsOutput_HeaderAndEmptyState_YieldNothing() + { + var stdout = """ + provider model context max-out thinking images + No models available. Set API keys in environment variables. + """; + + Assert.Empty(CavemanCodeModelListProbe.ParseModelsOutput(stdout)); + } + + [Fact] + public void ParseModelsOutput_SingleSpacedNoise_YieldsNothing() + { + // Columns are joined with 2+ spaces, so single-spaced process noise + // (npm chatter, log lines) never yields two columns. + var stdout = """ + added 241 packages in 1m + INFO: cache hit for registry + Loading providers... + """; + + Assert.Empty(CavemanCodeModelListProbe.ParseModelsOutput(stdout)); + } + + [Fact] + public void ParseModelsOutput_TruncatesAtMaxModelIds() + { + var lines = Enumerable.Range(0, CavemanCodeModelListProbe.MaxModelIds + 10) + .Select(i => $"openai model{i} 400K 128K yes yes"); + var stdout = string.Join('\n', lines); + + var ids = CavemanCodeModelListProbe.ParseModelsOutput(stdout); + + Assert.Equal(CavemanCodeModelListProbe.MaxModelIds, ids.Count); + } + + [Fact] + public async Task GetModelListAsync_NonZeroExit_Fails() + { + var probe = new CavemanCodeModelListProbe(new StubCavemanCodeCliRunner(1, "out", "boom")); + + var result = await probe.GetModelListAsync(CancellationToken.None); + + Assert.NotNull(result.FailureReason); + Assert.Empty(result.ModelIds); + } + + [Fact] + public async Task GetModelListAsync_Exit127_FailsAsNotFound() + { + var probe = new CavemanCodeModelListProbe(new StubCavemanCodeCliRunner(127, "", "")); + + var result = await probe.GetModelListAsync(CancellationToken.None); + + Assert.Equal("caveman-code CLI not found", result.FailureReason); + } + + [Fact] + public async Task GetModelListAsync_FileNotFoundThrow_MapsToNotFound() + { + var probe = new CavemanCodeModelListProbe(new ThrowingCavemanCodeCliRunner(new FileNotFoundException())); + + var result = await probe.GetModelListAsync(CancellationToken.None); + + Assert.Equal("caveman-code CLI not found", result.FailureReason); + } + + [Fact] + public async Task GetModelListAsync_EnoentThrow_MapsToNotFound() + { + var probe = new CavemanCodeModelListProbe( + new ThrowingCavemanCodeCliRunner(new Win32Exception(2, "No such file or directory"))); + + var result = await probe.GetModelListAsync(CancellationToken.None); + + Assert.Equal("caveman-code CLI not found", result.FailureReason); + } + + [Fact] + public async Task GetModelListAsync_Success_ParsesFixture() + { + var stdout = await File.ReadAllTextAsync(FixturePath); + var probe = new CavemanCodeModelListProbe(new StubCavemanCodeCliRunner(0, stdout, "")); + + var result = await probe.GetModelListAsync(CancellationToken.None); + + Assert.Null(result.FailureReason); + Assert.Contains("openai/gpt-5.5", result.ModelIds); + Assert.Contains("anthropic/claude-opus-4-6", result.ModelIds); + } + + [Fact] + public async Task DefaultCliRunner_StartFailed_MapsToFailureResult() + { + var process = new StartFailedProcessRunner(); + var runner = new DefaultCavemanCodeCliRunner( + process, + environment: new Dictionary()); + + var run = await runner.RunListModelsAsync("caveman-code", CancellationToken.None); + + Assert.Equal(new[] { "caveman-code", "--list-models" }, process.SeenArgv); + Assert.Equal(1, run.ExitCode); + Assert.Equal("", run.Stdout); + Assert.Equal("", run.Stderr); + } + + [Fact] + public async Task GetModelListAsync_BinaryMissing_FailsGracefullyThroughRealRunner() + { + // Real runner + probe wiring with only the process boundary stubbed: + // a missing binary must surface as a probe failure, never a throw + // and never an empty success. + var probe = new CavemanCodeModelListProbe( + new DefaultCavemanCodeCliRunner( + new StartFailedProcessRunner(), + environment: new Dictionary())); + + var result = await probe.GetModelListAsync(CancellationToken.None); + + Assert.NotNull(result.FailureReason); + Assert.Empty(result.ModelIds); + } + + private sealed class StartFailedProcessRunner : IProcessRunner + { + public IReadOnlyList SeenArgv { get; private set; } = Array.Empty(); + + public Task RunAsync( + IReadOnlyList argv, + string? stdin, + CancellationToken ct, + Action? stdoutChunkCallback = null, + Action? stderrChunkCallback = null, + int? maxStdoutBytes = null, + int? maxStderrBytes = null, + IReadOnlyDictionary? environment = null, + bool killOnOutputLimit = true) + { + SeenArgv = argv; + return Task.FromResult(new ProcessRunResult(1, "", "", StartFailed: true)); + } + } + + private sealed class StubCavemanCodeCliRunner(int exitCode, string stdout, string stderr) : ICavemanCodeCliRunner + { + public Task RunListModelsAsync(string binary, CancellationToken ct) => + Task.FromResult(new CavemanCodeCliRunResult(exitCode, stdout, stderr)); + } + + private sealed class ThrowingCavemanCodeCliRunner(Exception ex) : ICavemanCodeCliRunner + { + public Task RunListModelsAsync(string binary, CancellationToken ct) => + throw ex; + } +} diff --git a/tests/CodeyBox.Tests/CavemanCodeQuotaFailureDetectorTests.cs b/tests/CodeyBox.Tests/CavemanCodeQuotaFailureDetectorTests.cs new file mode 100644 index 000000000..a0011165e --- /dev/null +++ b/tests/CodeyBox.Tests/CavemanCodeQuotaFailureDetectorTests.cs @@ -0,0 +1,106 @@ +using CodeyBox.Agents; +using CodeyBox.Agents.CavemanCode; +using CodeyBox.Core; + +namespace CodeyBox.Tests; + +/// +/// Unit tests for . Every row +/// pins a production pattern to the exact stderr/stdout shape that motivated +/// it; removing or mistyping a pattern breaks the corresponding row. +/// +public sealed class CavemanCodeQuotaFailureDetectorTests +{ + private readonly CavemanCodeQuotaFailureDetector _detector = new(); + + [Fact] + public void Kind_IsCavemanCode() + { + Assert.Equal(AgentKind.CavemanCode, _detector.Kind); + } + + [Fact] + public void Detect_NullStreams_ReturnsNull() + { + Assert.Null(_detector.Detect(null, null)); + Assert.Null(_detector.Detect("", "")); + } + + // --- Unauthorized: missing-key output observed live (0.65.2) ------------ + + [Fact] + public void Detect_NoApiKeyFound_ClassifiesUnauthorized() + { + // Exact first line of the live keyless run; stdout carries it because + // the CLI exits 0 on missing auth. + var detection = _detector.Detect(stderr: null, stdout: "No API key found for unknown."); + + Assert.NotNull(detection); + Assert.Equal(QuotaFailureKind.Unauthorized, detection!.Kind); + } + + [Fact] + public void Detect_SetApiKeyEnvHint_ClassifiesUnauthorized() + { + var detection = _detector.Detect( + stderr: "Use /login or set an API key environment variable.", + stdout: null); + + Assert.NotNull(detection); + Assert.Equal(QuotaFailureKind.Unauthorized, detection!.Kind); + } + + [Fact] + public void Detect_BareUnauthorizedWord_DoesNotClassify() + { + // The bare word appears in code under review (access-control prose); + // only the HTTP-anchored rows may match. + Assert.Null(_detector.Detect("discusses Unauthorized access in the handler", null)); + } + + // --- RateLimitExceeded: transient provider refusals --------------------- + + public static IEnumerable RateLimitExceededSamples() + { + // Shared anchored 429 rows (mirrors the Copilot/opencode detectors). + yield return new object[] { "Error: HTTP 429 from model endpoint", "HTTP 429" }; + yield return new object[] { "Status: 429 Too Many Requests, retry later", "429 Too Many Requests" }; + yield return new object[] { "Upstream: rate_limit_exceeded, backing off", "rate_limit_exceeded" }; + yield return new object[] { "Provider says: Too Many Requests", "Too Many Requests" }; + // Provider error code matched by the CLI's own retry classifier. + yield return new object[] { "Request failed with overloaded_error, retrying", "overloaded_error" }; + } + + [Theory] + [MemberData(nameof(RateLimitExceededSamples))] + public void Detect_RateLimitShapes_AllClassifyAsRateLimitExceeded(string stderr, string pattern) + { + var detection = _detector.Detect(stderr: stderr, stdout: null); + + Assert.NotNull(detection); + Assert.Equal(QuotaFailureKind.RateLimitExceeded, detection!.Kind); + Assert.Contains(pattern, stderr, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public void Detect_BareOverloadedWord_DoesNotClassify() + { + // Suffix-anchored: prose about an "overloaded" server must not bench + // the agent; only the provider's overloaded_error code matches. + Assert.Null(_detector.Detect("the overloaded server dropped connections", null)); + } + + [Fact] + public void Detect_Bare429Number_DoesNotClassify() + { + // A bare number in reviewed code (retry counts, line numbers) must + // not gate dispatch — rows stay anchored with companion text. + Assert.Null(_detector.Detect("retry 429 times per the spec", null)); + } + + [Fact] + public void Detect_UnrelatedError_ReturnsNull() + { + Assert.Null(_detector.Detect("Error: something entirely different broke", "plain output")); + } +} diff --git a/tests/CodeyBox.Tests/CavemanCodeSmokeProbeTests.cs b/tests/CodeyBox.Tests/CavemanCodeSmokeProbeTests.cs new file mode 100644 index 000000000..20cdfb163 --- /dev/null +++ b/tests/CodeyBox.Tests/CavemanCodeSmokeProbeTests.cs @@ -0,0 +1,98 @@ +using CodeyBox.Agents.CavemanCode; +using CodeyBox.Core; + +namespace CodeyBox.Tests; + +public sealed class CavemanCodeSmokeProbeTests +{ + [Fact] + public async Task SmokeTest_NoKeys_ReturnsFail() + { + var probe = new CavemanCodeSmokeProbe(); + var cred = new AgentCredential(AgentKind.CavemanCode, + new Dictionary(), + new Dictionary()); + + var result = await probe.SmokeTestAsync(cred, CancellationToken.None); + + Assert.False(result.Ok); + Assert.Equal("no provider API key in credential bundle", result.FailureReason); + } + + [Fact] + public async Task SmokeTest_EmptyKeyValue_ReturnsFail() + { + var probe = new CavemanCodeSmokeProbe(); + var cred = new AgentCredential(AgentKind.CavemanCode, + new Dictionary { ["OPENAI_API_KEY"] = "" }, + new Dictionary()); + + var result = await probe.SmokeTestAsync(cred, CancellationToken.None); + + Assert.False(result.Ok); + } + + [Theory] + [InlineData("ANTHROPIC_API_KEY")] + [InlineData("OPENAI_API_KEY")] + [InlineData("GEMINI_API_KEY")] + [InlineData("OPENROUTER_API_KEY")] + public async Task SmokeTest_AnyProviderKey_ReturnsOk(string variable) + { + var probe = new CavemanCodeSmokeProbe(); + var cred = new AgentCredential(AgentKind.CavemanCode, + new Dictionary { [variable] = "test-key" }, + new Dictionary()); + + var result = await probe.SmokeTestAsync(cred, CancellationToken.None); + + Assert.True(result.Ok); + Assert.Null(result.FailureReason); + } +} + +public sealed class CavemanCodeInVmSmokeProbeTests +{ + private readonly CavemanCodeInVmSmokeProbe _probe = new(); + + [Fact] + public void Kind_IsCavemanCode() + { + Assert.Equal(AgentKind.CavemanCode, _probe.Kind); + } + + [Fact] + public void BuildSteps_NoCredential_ReturnsVersionCheckOnly() + { + var steps = _probe.BuildSteps(credential: null); + + var argv = Assert.Single(steps).Argv; + Assert.Equal(["caveman-code", "--version"], argv); + } + + [Fact] + public void BuildSteps_KeylessCredential_ReturnsVersionCheckOnly() + { + var cred = new AgentCredential(AgentKind.CavemanCode, + new Dictionary { ["SOME_OTHER_VAR"] = "x" }, + new Dictionary()); + + var steps = _probe.BuildSteps(cred); + + Assert.Single(steps); + } + + [Fact] + public void BuildSteps_WithProviderKey_AddsListModelsStep() + { + var cred = new AgentCredential(AgentKind.CavemanCode, + new Dictionary { ["OPENAI_API_KEY"] = "sk-test" }, + new Dictionary()); + + var steps = _probe.BuildSteps(cred); + + Assert.Equal(2, steps.Count); + Assert.Equal(["caveman-code", "--version"], steps[0].Argv); + Assert.Equal(["caveman-code", "--list-models"], steps[1].Argv); + } +} diff --git a/tests/CodeyBox.Tests/CavemanCodeStreamParserTests.cs b/tests/CodeyBox.Tests/CavemanCodeStreamParserTests.cs new file mode 100644 index 000000000..b043878df --- /dev/null +++ b/tests/CodeyBox.Tests/CavemanCodeStreamParserTests.cs @@ -0,0 +1,49 @@ +using System.Text.Json; +using CodeyBox.Agents.CavemanCode; +using CodeyBox.Core; + +namespace CodeyBox.Tests; + +public sealed class CavemanCodeStreamParserTests +{ + [Fact] + public void TryClaim_JsonLine_ReturnsFalse() + { + // Plaintext -p output: the parser claims nothing by shape so the + // plaintext-fallback summariser handles the run (opencode parity). + var parser = new CavemanCodeStreamParser(); + using var doc = JsonDocument.Parse("""{"type":"message","content":"hi"}"""); + + Assert.False(parser.TryClaim(doc.RootElement)); + } +} + +public sealed class DefaultCavemanCodeCliRunnerTests +{ + [Fact] + public async Task RunListModelsAsync_StartFailed_MapsToExitOneEmpty() + { + var runner = new DefaultCavemanCodeCliRunner(new StartFailingProcessRunner()); + + var result = await runner.RunListModelsAsync("caveman-code", CancellationToken.None); + + Assert.Equal(1, result.ExitCode); + Assert.Equal("", result.Stdout); + Assert.Equal("", result.Stderr); + } + + private sealed class StartFailingProcessRunner : CodeyBox.HostProcess.IProcessRunner + { + public Task RunAsync( + IReadOnlyList argv, + string? stdin, + CancellationToken ct, + Action? stdoutChunkCallback = null, + Action? stderrChunkCallback = null, + int? maxStdoutBytes = null, + int? maxStderrBytes = null, + IReadOnlyDictionary? environment = null, + bool killOnOutputLimit = true) => + Task.FromResult(new CodeyBox.HostProcess.ProcessRunResult(1, "", "", StartFailed: true)); + } +} diff --git a/tests/CodeyBox.Tests/CodeyBox.Tests.csproj b/tests/CodeyBox.Tests/CodeyBox.Tests.csproj index 83d747dbd..d3610a6be 100644 --- a/tests/CodeyBox.Tests/CodeyBox.Tests.csproj +++ b/tests/CodeyBox.Tests/CodeyBox.Tests.csproj @@ -71,6 +71,7 @@ + diff --git a/tests/CodeyBox.Tests/Fixtures/CavemanCode/caveman-code-list-models.redacted.txt b/tests/CodeyBox.Tests/Fixtures/CavemanCode/caveman-code-list-models.redacted.txt new file mode 100644 index 000000000..f8ea2f417 --- /dev/null +++ b/tests/CodeyBox.Tests/Fixtures/CavemanCode/caveman-code-list-models.redacted.txt @@ -0,0 +1,6 @@ +provider model context max-out thinking images +anthropic claude-haiku-4-5 200K 64K yes yes +anthropic claude-opus-4-6 1M 128K yes yes +anthropic claude-sonnet-4-6 1M 64K yes yes +openai gpt-5-codex 400K 128K yes yes +openai gpt-5.5 1.1M 128K yes yes