api.qa/vitest@1 — executable suites runner (Worker Loader, flag-held) - #7
Conversation
…te runner, grading wire-in
Implements the runner phase of AXP 0.7.0 / apis-ax-axp@2.4.0 (spec digest
dd3e5941…): the executable published-test-suite dialect the corrected spec
ratified beside the declarative api.qa/suite@1 path.
ONE SHARED HARNESS (A.8.6.2, normative parity). src/exec/vitest-subset.mjs is
the single implementation of the guaranteed vitest subset (describe/it/expect
+ async, rejects/resolves, .not; snapshots/vi/unknown matchers fail BY NAME).
scripts/gen-vitest-subset.mjs carries its exact bytes into a string constant so
the hosted isolate's module map and the local runner instantiate byte-identical
code; a pinning test fails on drift. Local==hosted by construction.
CODE FIELDS, ONE ARTIFACT, ONE DIGEST (A.8.6.1). suite-doc.ts gains
parseExecSuiteDocument: the additive suite@1 extension carrying `tests`
(required) + `module` (optional) as string members; types.ts carries them plus
the environment `sandbox` consent flag. One fetch, one buffer,
hash-then-instantiate — digest fail-closed before anything runs.
THE THREE COLLAPSED CHANNELS (A.8.6.6). test-suite.ts gateVitestSuiteCard /
gateVitestSuiteDocument / gateVitestModuleArtifact: inline document strings,
{url,digest} natively-served ESM (off-origin module-CDN allowed — the digest,
never the host, is the authority), and {package,version,digest} as an IDENTITY
ASSERTION recorded but never adjudicated (the registry is never contacted). The
widened seam { url?, package?, version?, export?, digest, environment?, runner? }
is read in discovery.ts; artifact kind is decided by the card, never by sniffing.
ISOLATE POSTURE (A.8.6.3). src/exec/runner.ts (Worker Loader) + dialect.ts:
zero ambient authority (env is always {}), a network floor that bars
metadata/link-local/loopback/RFC1918/CGNAT/ULA/estate-internal ONLY and permits
all other external egress (cross-estate composition is a feature; graded-origin
scoping rejected), a metered circuit-breaker (300s wall / 60s CPU, billed,
account-raisable), seeded Math.random, mutating verbs only where the environment
declares sandbox:true, and fail-closed totality (a caught floor refusal still
fails the run). worker_loaders is feature-detected: absent binding (or absent
egress gateway) yields a typed runner-unavailable outcome, never a crash or a
silent pass; wrangler.jsonc documents the flag-held binding.
GRADING + ATTESTATION (A.8.5.2 / A.8.6.5). checks.ts arms
`published-test-suite` on the vitest@1 runner and judges purely from the
recorded typed run outcome (declared => run => must pass; undeclared => skip).
The verdict carries executed digest, discriminated kind, environment + sandbox,
seed, applied breaker limits + elapsed, folded rows+tests, and the npm
coordinate where asserted. src/pinned.ts pins AXP_PINNED_SPEC to the final
2.4.0 digest dd3e5941….
LOCAL PARITY. cli/index.ts gains `vitest <artifact> --target …` running the
dialect through the same shared harness; exits non-zero on any failing test/row
and fail-closed on a --expect-digest mismatch.
Tests: harness subset behaviour, digest fail-closed, network floor
(metadata/RFC1918 blocked, external allowed), the metered breaker,
declared/undeclared arming, runner-unavailable, an end-to-end fixture, and the
CLI verb. Full suite 1226 passing; typecheck + build green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The local runner's default realFetch closure resolved globalThis.fetch at call time — AFTER run() had already swapped globalThis.fetch to the gated fetch — so every egress recursed gate->global->gate to a blown stack. Only the CLI vitest verb rode the default (tests inject io.fetch), so the bug was invisible in-repo and surfaced on the first adopter run (apis.vin). Bind the ambient fetch before the swap and hand THAT to the gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…band gateway record + drain (A.8.6.3)
P1 (attested path): the hosted isolate could SWALLOW a network-floor
refusal. gatewayFetch recorded the violation into a locally-created,
discarded array and handed the isolate a plain catchable 403; the entry's
fetch wrapper recorded only VERB violations. A suite that try/caught the
refused fetch (or merely inspected the 403) passed hosted while the same
bytes failed locally — violating both local==hosted parity and A.8.6.3
fail-closed totality (apis-ax-axp@2.4.0).
Now the refusal is recorded where suite code can never reach it, and the
runner hard-fails the verdict regardless of what the suite caught:
- gatewayFetch takes a caller-owned violations sink and stamps refused
responses with the x-apiqa-gateway marker ("violation" | "error");
- createOutboundGateway pairs the globalOutbound fetch handler with
drainViolations() — the parent-memory, out-of-band record;
- workerLoaderExecRunner drains that record on EVERY exit path (body,
isolate error, crash; cleared on breaker trip) and folds it into the
outcome ahead of anything the isolate reported; an unreadable record
fails CLOSED by a named reason (GATEWAY_RECORD_UNREADABLE);
- the isolate entry re-throws a marked 403 with the gateway's own reason
and records it in the run's violation list — the same throw shape the
local gated fetch gives — with Response/JSON.stringify captured before
any suite byte runs so the body cannot be forged in-isolate.
Tests: executed-module-map simulation of the hosted path proves (a) a
try/caught refused fetch still FAILS the run with the floor reason, and
(b) local and hosted produce IDENTICAL verdicts and reason strings for
the same suite (swallowed floor refusal, clean pass, verb gate); plus
the forged-all-green-body case and the unreadable-record fail-closed case.
P3: the gatewayFetch comment claimed "the verb/floor policy ALSO fails
the run via the in-isolate wrapper's violation record" — untrue before
this change. Comments in runner.ts / worker.ts / wrangler.jsonc now state
exactly what each half guarantees: gatewayFetch refuses + records + marks;
the RUNNER enforces the run-level fail-closed verdict via the drained sink.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9511d9bd32
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const id = `vitest1:${req.digest ?? 'inline'}:${HARNESS_VERSION}` | ||
| const stub = loader.get(id, () => buildWorkerCode(req, opts.outbound)) |
There was a problem hiding this comment.
When the same artifact digest is verified more than once, WorkerLoader.get() may reuse the worker cached under this ID, but buildWorkerCode() embeds request-specific values such as the origin, environment variables, seed, sandbox flag, and export name. A reused worker therefore runs with the first request's configuration; moreover, its fixed ./suite-tests.mjs import is already evaluated, so a subsequent fetch can register zero tests. Use the one-shot loader API or a per-invocation ID rather than treating artifact-only caching as single-use isolation.
Useful? React with 👍 / 👎.
| const breaker = new Promise<'breaker'>((resolve) => { | ||
| timer = setTimeout(() => resolve('breaker'), wallMs) | ||
| }) | ||
| const raced = await Promise.race([execute(), breaker]).finally(() => clearTimeout(timer)) |
There was a problem hiding this comment.
If a test remains pending past wallMs, Promise.race() only stops waiting for execute(); it does not cancel it. The runner then returns through finally, restores the ambient Node fetch, and leaves the test promise running, so a delayed test can subsequently perform floor-blocked requests through the ungated fetch (and keep the CLI process alive). The timeout must terminate an isolated execution context rather than returning while suite code is still active.
Useful? React with 👍 / 👎.
| if (!isModule) { | ||
| try { | ||
| doc = parseExecSuiteDocument(text) |
There was a problem hiding this comment.
For document artifacts, this path only calls parseExecSuiteDocument() and never applies gateVitestSuiteDocument(), while the hosted path rejects non-endpoint rows and any declarative method other than GET/HEAD. The later local verifySuite() call is consent mode and permits writes, so a document containing a POST row can mutate the target and pass locally even though hosted verification refuses the artifact before execution. Reuse the same document gate before starting either part of the local run.
Useful? React with 👍 / 👎.
| if (!target) return die('vitest needs a target: --target <origin> (or a string `baseUrl` var in the selected environment)') | ||
| const runSeed = seed ?? (Math.floor(Math.random() * 0xffffffff) >>> 0) | ||
|
|
||
| const outcome: ExecRunOutcome = await localExecRunner().run({ |
There was a problem hiding this comment.
The local CLI starts executable tests here and only runs declarative rows afterward, whereas observeVitestSuite() runs rows before calling execRunner.run(). For a sandbox suite whose executable test creates, updates, or deletes state observed by a row, local and hosted runs therefore inspect different target states and can produce opposite verdicts under the same digest and seed. Preserve the hosted ordering in the CLI before claiming local/hosted parity.
Useful? React with 👍 / 👎.
| const importFresh = (source: string, map: Record<string, string>, tag: string) => | ||
| // The trailing comment makes every run's URL unique: ES module | ||
| // instances are cached BY URL, and a re-run of the same bytes must | ||
| // re-register its tests, not reuse a spent registration. | ||
| import(/* @vite-ignore */ dataModuleUrl(`${rewriteSpecifiers(source, map)}\n//# ${runId}:${tag}`)) |
There was a problem hiding this comment.
Importing the suite as a data: module in the CLI's own Node realm exposes globals that do not exist in the hosted isolate, including process. A suite can read process.env and exfiltrate CI credentials through the intentionally permitted public egress without using any forbidden import, and ordinary tests that inspect Node globals also diverge between local and hosted runs. Execute local artifacts in a restricted isolate or worker instead of relying on textual import validation.
Useful? React with 👍 / 👎.
| // // own network access. Prefer a same-isolate loopback entrypoint | ||
| // // (`ctx.exports.SuiteGateway`) over a separate worker, so | ||
| // // `drainViolations` is actually drainable by the runner: | ||
| // "services": [ { "binding": "SUITE_OUTBOUND", "service": "api-qa", "entrypoint": "SuiteGateway" } ], |
There was a problem hiding this comment.
The documented provisioning configuration binds SUITE_OUTBOUND to the named SuiteGateway entrypoint, but a repo-wide search finds no such export or WorkerEntrypoint implementation in src; worker.ts only exports the default worker entry. Uncommenting the advertised bindings therefore cannot provision the runner, and there is no deployed wrapper connecting createOutboundGateway() and drainViolations() to this service binding. Add the entrypoint before presenting this as the enablement path.
Useful? React with 👍 / 👎.
1 participant
api.qa/vitest@1 — executable suites runner (Worker Loader, flag-held)
What changed
dd3e5941…coordinated with the AXP 0.7.0 spec.Gates
🤖 Generated with Claude Code