Say what a docs call actually observed, and show what was asked

[leggetter]

The task

#83 — docs-call reporting counts results we never saw as evidence. Widened, on
review, to cover the other half of the same error and to make the observable part
of a search available for analysis.

What changed

apps/framework/lib/docs-reach.ts — one pure function classifying a docs call
four ways, and a line of prose per bucket for whatever prints them.

bucket	meaning	produced by
read	page text reached the agent	web_fetch, shell_fetch
hits	a list of links, no page text	Claude Code's WebSearch
none	a result we saw, carrying nothing	any source
unobserved	a hosted search whose hits never reach us	Codex's web_search

Two fields do the work, and both are needed. resultChars separates we saw a
result and it was empty (0) from there was no result to see (omitted).
hasContent separates page text from a list of links — docs-results.ts already
sets it to false for Claude's WebSearch for exactly this reason.

--queries on report-results prints what each arm asked and what came back,
grouped by scenario so both arms of a pair sit together.

The error, in both directions and both agents

On results/runs/2026-09-14T10-36-39-000Z.json:

claude-code-sonnet-5-no-skills    46 calls  read 36  hits 10  none 0  unobserved   0   78% read
codex-gpt-5.4-mini-no-skills     205 calls  read  0  hits  0  none 0  unobserved 205    0% read
codex-gpt-5.4-mini                23 calls  read 14  hits  0  none 0  unobserved   9   61% read
claude-code-sonnet-5              22 calls  read 22  hits  0  none 0  unobserved   0  100% read

• Codex: all 254 searches were unobserved. 216 were published as empty — a
  search that failed — and 38 as reached a page, because a url-shaped query is
  recorded as the page it probably opened.
• Claude: 10 of the baseline arm's 46 credited page reads were hit lists.
  Titles and urls. It was credited for seeing a list of links.

none is zero across this snapshot. The bucket is still right to keep — a
WebSearch with no hits lands there — it just says nothing here was an observed
failure, which is the opposite of what we published.

Why the queries view earns its place

For a hosted search the query is the only observable there is, and it turns out to
be the half that answers the question. verification-002, 14 September:

codex-gpt-5.4-mini-no-skills  FAILED
  unobserved  web_search  site:hookdeck.com/docs sources ElevenLabs Hookdeck
  unobserved  web_search  site:hookdeck.com/docs/sources ElevenLabs source type Hookdeck ElevenLabs-Signature

It then built a generic WEBHOOK source with hand-rolled HMAC against
ElevenLabs-Signature and failed the first check. The arm with the skill ran
--source-type ELEVENLABS and passed. The pass rate cannot show that; two queries
and their outcome can.

Off by default, because it is long.

Why widen rather than file a follow-up

The hits over-credit is the same defect as the unobserved one — crediting the
reporter's assumption instead of an observation — in the same function, on the same
line. Splitting it across two PRs would have meant changing that line twice and
leaving a published number wrong in between.

Verified

pnpm format:check   biome 150 files, no fixes; prettier all matched
pnpm typecheck      framework + web, clean
pnpm check          21 test files, 234 tests, all passed  (was 227; 7 new)
pnpm -r build       done

Plus both report modes against a real snapshot, output above. Nothing was already
failing. pnpm install needs /opt/homebrew/bin ahead of the asdf shims.

Tests

apps/framework/test/docs-reach.test.ts, seven cases, in the place this repo
already puts them — test/ against lib/, never against scripts/, which is why
the classifier is a lib module rather than a private function in the script.

The two that matter are the ones that were published wrong: a hit list is not a
read, and a url-shaped query with no result is not a page read.

Not done

• Documentation access differs by agent and by arm, and the search path mostly returns nothing #61's table is not re-derived here. Its headline — "the search path mostly
  returns nothing", 58% — rests on the old bucket. Re-deriving it is work on that
  issue, which also has a second half this does not touch: the two arms really do
  differ in tool mix. Commented there.
• No published number changes. docs.calls is diagnostic; the scoreboard does
  not read it.
• --queries prints every call rather than sampling. It is opt-in and piped to
  grep in practice; a filter is easy to add if anyone wants one.
• Nothing attempts to capture hosted search results. They never leave the
  provider, so there is nothing to capture. That is the limitation this makes
  visible rather than one it fixes.