test: smoke the agent against a real local model, gating the publish

[siarheidudko]

Why here, not nightly

A nightly run reports at 3am on yesterday's commit and nobody reads it. The question this layer answers — does the agent still work with a model that has opinions? — matters at the moment something is about to be published, so release.yml now needs: it. It also runs on PRs touching package.json / package-lock.json, which is where upstream drift actually arrives (the autoupdate flow's PRs land there).

Nightly's one unique property is catching drift with no commits at all; that is already covered by the autoupdate flow opening PRs.

What

• tests/live-model.test.ts — the same loop as the scripted integration suite, driven by a real model behind any OpenAI-compatible endpoint. Skipped unless AGENT_LIVE_MODEL_URL is set, so npm test is unaffected (172 tests, 1 skipped locally).
• .github/workflows/live-model.yml — pulls a pinned llama.cpp server image and a pinned GGUF (cached), serves it on the runner, runs the test. No hosted API, no key.
• tests/helpers/mcp-server.ts — a secret tool returning a per-run random value.
• release.yml gains a live-model job that release depends on.

Assertions are mechanics only: the loop reached a final answer, the plan parsed, the model drove the MCP tool, and the tool's result reached the final text. Asserting wording or plan shape would be asserting the model's judgement, which changes with every weight and sampler — that is how a suite ends up disabled. Sampling is greedy with a fixed seed, and the step retries once so one stray run isn't treated as signal.

It is a blocking gate, and the model floor is measured

An earlier revision of this PR shipped the job advisory (continue-on-error: true) because the tool-calling half was unverified. It has since been run for real, and both the flag and the model are now decided by measurement rather than caution.

What the runs showed, all on the runtime this workflow actually uses (llama-server --jinja, greedy):

	plan	tool call	answer
Qwen2.5-1.5B	parses	never	invented a code — one run reported a tool's name as the secret
Qwen2.5-3B	parses	✅ files__secret	contained the real zq-… value

So the pin is 3B, and continue-on-error is gone. The earlier negative result for the 1.5B was partly my own instrumentation: the first attempt went through llama-cpp-python's chatml-function-calling shim, and on the real llama-server the 1.5B does emit tool calls for a direct single-tool request. It still doesn't do it through the agent, which is what matters here.

Two findings worth keeping, both written into comments next to the code they explain:

• The task must be unanswerable without the tool. "Echo hello" lets the planner take its answer-directly branch and finish with a plausible sentence and no tool call. Hence the secret tool: a value only the server knows.
• It must read as a question, not an order. "Call the secret tool and report the code" still lost to the answer-directly branch on the 3B, which then invented a code. "What is the server-side secret code?" matches the planner's own rule about information the tools can retrieve, and the tool gets called. The planner prompt's canned step "Answer the user directly…" is strongly attractive to small models — useful to know beyond this test.

Note on release.yml

CLAUDE.md says not to touch that file unless explicitly asked. This change was explicitly requested (gate the release on a real-model run); the edit is two lines — a live-model job that calls the reusable workflow, and needs: live-model on the publish job. The quirks section in CLAUDE.md records why, so a future session doesn't "fix" it away.

typecheck, format:check, build, test are green (171 pass, 1 skipped), and live-model.test.ts passed against the 3B in 10.3s.

---

Generated by Claude Code