fix(verifier): declarative suite rows read up to 4 MiB and truncation is judged honestly - #10
Conversation
… is judged honestly A real verifier bug (the apis.vin/listings false-fail): declarative suite rows — the card-declared suite@1 sub-run (runSuiteRows) and pinned/ verifySuite endpoint requirements — were fetched under the Observer's 256 KiB discovery-surface default cap. A legitimate ~308 KB valid-JSON API response was severed mid-token, readCapped's truncation never reached the Evidence, and judgeExpect reported the lie "body is not JSON" although the target answered 200 with a perfectly valid document. Two changes, neither making the grader lenient: - SUITE_ROW_MAX_BODY_BYTES (4_194_304 — the exec module-artifact ceiling): the runSuiteRows child observer and the pinned-mode observer now read row responses up to 4 MiB. Rows fetch sequentially, so peak retained memory stays one body <= the cap. - Evidence.truncated: readCapped now reports whether it cut the read (at the streaming cap, or refusing a declared-oversized Content-Length), the observer records it, and judgeExpect fails a truncated body with the honest "body too large: truncated at N bytes by the verifier read cap" — never a misleading "not JSON", and never judging partial data that happens to parse. captureInto refuses to bind vars out of a cut body. Both directions proven in test/suite-row-body-cap.test.ts: a ~308 KB and a 1 MiB valid JSON row PASS; a genuinely non-JSON body still FAILS "body is not JSON"; a body beyond even the raised cap FAILS with the truncation reason. The field is optional (absent, never false) so stored evidence bundles replay byte-identically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c8085c883a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // default; under that default a large VALID body was severed mid-token and | ||
| // misjudged "body is not JSON". Beyond even this cap, `Evidence.truncated` | ||
| // makes judgeExpect fail with the honest truncation reason. | ||
| const sub = observer.child({ allowWrites: false, budget, maxBodyBytes: SUITE_ROW_MAX_BODY_BYTES }) |
There was a problem hiding this comment.
When a card-declared suite uses the allowed 25 requirements and each endpoint returns close to the new 4 MiB cap, every response remains stored in sub.items and is then retained in the parent evidence bundle. Sequential fetching therefore does not keep peak retained data to one response: an untrusted target can produce roughly 100 MiB of body evidence before the bundle is digested and serialized, potentially exhausting the verifier. Add an aggregate byte budget or otherwise account for all retained row bodies.
Useful? React with 👍 / 👎.
| // card-declared path uses (a legitimate large JSON response must not be | ||
| // severed mid-token and misjudged "not JSON"). An explicit caller cap | ||
| // still wins. | ||
| maxBodyBytes: opts.maxBodyBytes ?? SUITE_ROW_MAX_BODY_BYTES, |
There was a problem hiding this comment.
In verifySuite/verifyPinnedSpec runs without an explicit maxBodyBytes, this same observer is passed to observeTarget at line 203 before the endpoint-requirement loop. Consequently all root variants, discovery documents, OpenAPI data, and card-directed probes now receive the 4 MiB cap, even though the change is intended only for endpoint rows; oversized discovery surfaces that were previously bounded at 256 KiB can now be retained and parsed. Use a dedicated observer or per-request cap for the endpoint requirements rather than raising the parent observer's cap.
Useful? React with 👍 / 👎.
| offset += chunk.byteLength | ||
| } | ||
| return new TextDecoder('utf-8').decode(buf) | ||
| return { text: new TextDecoder('utf-8').decode(buf), truncated } |
There was a problem hiding this comment.
For a streaming response whose final chunk exactly fills maxBodyBytes, the existing value.byteLength >= remaining branch sets truncated = true even though no byte was discarded. Returning that flag now records Evidence.truncated, so a complete valid JSON body of exactly 4 MiB—or exactly 256 KiB for default observers—fails schema/path expectations as oversized, whereas it passed before this change. Distinguish an exact complete read from an actual over-cap read, using the declared length when available or checking for EOF.
Useful? React with 👍 / 👎.
1 participant
The bug (a real verifier false-fail)
Declarative suite rows — the card-declared
suite@1sub-run (runSuiteRows, src/discovery.ts) and pinned/verifySuiteendpoint requirements (src/pinned.ts) — were fetched under the Observer's 256 KiB discovery-surface default body cap. A real API endpoint that legitimately returns hundreds of KB of VALID JSON (apis.vin/listings, ~308 KB) had its body severed mid-token,readCapped's truncation never reached the Evidence, andjudgeExpectreported the lie "body is not JSON" although the target answered 200 with a perfectly valid document.The fix
SUITE_ROW_MAX_BODY_BYTES = 4_194_304(4 MiB — the exec module-artifact ceiling,EXEC_MAX_MODULE_BYTES): therunSuiteRowschild observer and the pinned-mode observer now read row responses up to 4 MiB. Rows fetch sequentially, so peak retained memory stays one body ≤ the cap. Exported from the package root alongside the other published caps.Evidence.truncated:readCappednow reports whether it cut the read (at the streaming cap, or refusing a declared-oversized Content-Length before reading), the observer records it on the Evidence, andjudgeExpectfails a truncated body with the honestbody too large: truncated at N bytes by the verifier read cap— never a misleading "not JSON", and never judging partial data that happens to parse.captureIntorefuses to bind vars out of a cut body. The field is optional (absent, neverfalse), so stored evidence bundles replay byte-identically.Not more lenient — proven both directions (test/suite-row-body-cap.test.ts)
verifySuitepath);body is not JSON;truncatedon both cut paths; a truncated-but-parseable body is refused; an untruncated severed body still fails "not JSON".Full suite: 44 files, 1267 passed / 4 skipped.
tsc --noEmitandnpm run buildclean.🤖 Generated with Claude Code