agent-coding-session: per-group token_usage is last-call-wins — token sums undercount and depend on sync granularity

[bdelanghe]

Evidence from a live session (path-claude-code-25095aad, claude-code 2.1.241):

• The doc covers the FULL conversation — 347 steps, last ingested step is line 4088 of the 4090-line transcript. Coverage is fine.
• But only 72 steps carry structural.token_usage, summing to 102,354 output tokens, while the transcript's per-API-call usage (529 unique assistant message ids) sums to 784,105.
• Same binary base earlier read 729,846 for the same session — because incremental sync (a path query every ~25s from a statusline) ingested fine-grained turn groups, each capturing a usage snapshot. A from-scratch rebuild of the same transcript makes coarse groups and keeps only the last call's usage per group.

So token sums from path query are a function of sync cadence, not of the session — two syncs of identical data disagree by 7×. For spend accounting the kind should either sum token_usage across the API calls inside a group at ingest, or store the per-call usage array so consumers can sum. (Found while reconciling toolpath against transcript + OTEL + proxy witnesses, which now agree with each other and not with path.)

[bdelanghe]

Root-caused on phobos against the live data. The headline finding overturns this issue's premise: the per-message accounting is correct — the losses happen in sync, not derive.

canonicalize_message_usage (toolpath-claude/src/provider.rs:568-599) and last_of_message (toolpath-convo/src/derive.rs:188-200) both bound their runs on the message id, so usage can never merge across two msg_… ids. Empirically, the cached doc for 25095aad has 72 distinct group_ids and exactly 72 usage-bearing steps — one per message. A jq re-implementation of view→canonicalize→derive over the current transcript reproduces the transcript's own total (901,213 output tokens) exactly. The math is right.

The 102k-vs-784k discrepancy is a stale cache fossil: the cached doc's last step (16:21:49.608Z) matches the manifest's modified to the millisecond, and its first step predates the current transcript's first surviving entry — the transcript was compacted/rewritten in place after that sync, and the doc froze. The real defects:

1. Sync is replace-not-merge and the source can shrink (path-cli/src/sync/engine.rs:262-280 → cache.rs:42-62, truncate(true)). Claude Code rewrites session files on compaction; every re-derive discards whatever the transcript no longer contains. This is the whole cadence dependence: frequent syncs fossilize more of the session before compaction eats it. Fix: before write_cached, load the existing doc and union steps by step.id (or minimally: refuse to shrink Σ token_usage and warn).
2. .orphaned-* stems are misclassified as chain successors (toolpath-claude/src/chain.rs:94-101), and then is_bridge_entry is applied to every entry instead of the segment's first (lib.rs:156-161) — the whole segment's turns and usage silently vanish while its bytes still count into the freshness stamp.
3. Path/cache ids truncate the session uuid to 8 chars (toolpath-convo/src/derive.rs:49-54) — two sessions sharing a prefix overwrite each other's doc.
4. Perf: one git show subprocess per Write tool-use (provider.rs:166-226) makes the biggest sessions the ones most likely to stay stale under a 25s query loop.

Test gaps: nothing covers a shrinking source, nothing asserts a re-derive never lowers Σ token_usage, and test_is_bridge_entry never sees an .orphaned-* stem.

Until #1 lands, the transcript sum stays spend ground truth and path stays a coverage witness — but note path query also sums docs whose source files no longer exist (cache is archive-not-mirror, cache.rs:117), so its totals are a superset across renames, not just an undercount.

[bdelanghe]

Re-verified against main @ b31f2c5 on 2026-08-27. Still open and still real — but the headline number no longer reproduces, and the reason is worth recording before someone spends a day chasing it.

The four causes are all still in the code, verbatim

#	cause	current state
1	sync is replace-not-merge	path-cli/src/cache.rs:57 still opts.write(true).truncate(true); no union-by-step.id anywhere before write_cached
2	.orphaned-* misclassified + is_bridge_entry over-applied	zero matches for orphaned in toolpath-claude; lib.rs:157 still calls it per-entry inside for entry in &convo.entries
3	ids truncate the session uuid to 8 chars	toolpath-convo/src/derive.rs:49 still view.id.chars().take(8)
4	one git show subprocess per Write	toolpath-claude/src/provider.rs:215 unchanged

The three named test gaps are also still absent: no shrinking-source test, no "a re-derive never lowers Σ token_usage" assertion, no .orphaned-* case in test_is_bridge_entry.

The 102k-vs-784k evidence is gone from disk

25095aad today:

doc          347 steps · 72 usage-bearing · 72 group_ids · Σ output 102,354
transcript   427 lines · 75 groups                       · Σ output 105,865

The transcript is 427 lines with 72 message ids — not the 4,090 lines / 529 message ids the issue measured. So the 784,105 figure can no longer be reproduced from this machine, which is itself consistent with the comment above: the source shrank and, because sync is replace-not-merge, the archive followed it down. The 7× ratio should be treated as unverifiable history, not as the current severity.

What is live: a small, systematic, perfectly reproducible loss

Three sessions, and the gap is identical in all three — exactly 3 groups and exactly 3,511 output tokens:

session	groups (transcript → doc)	Σ output (transcript → doc)
25095aad	75 → 72	105,865 → 102,354
50357658	79 → 76	110,111 → 106,600
5217d8e1	59 → 56	90,747 → 87,236

The missing groups are msg_011CeMqkikywmqz9PuXLNk94 (1,148), msg_011CeMqoMX8Mk2wMbPJ6YJ39 (1,557) and msg_011CeMr4V9w4Zqq9gfGgRchp (806) — summing to exactly 3,511. All three transcripts contain the same three messages (a replayed prefix), which is why the delta is constant rather than proportional.

They are ordinary claude-opus-5 assistant messages with stop_reason: tool_use, not sidechains, and one of them carries visible prose. Reproduction:

D=~/.toolpath/documents/claude-path-claude-code-25095aad.json
S=~/.claude/projects/<slug>/25095aad-9e12-472d-80b1-34049009561f.jsonl

# groups the document carries
jq -r '.paths[].steps[]|.change|to_entries[]|.value.structural
       |select(.token_usage!=null)|.group_id' "$D" | sort -u > /tmp/doc

# groups the transcript carries
jq -r 'select(.type=="assistant" and .message.usage)|.message.id' "$S" | sort -u > /tmp/tx

comm -23 /tmp/tx /tmp/doc      # -> the three ids above

One explanation ruled out

is_bridge_entry (cause 2) is not the mechanism for this residual. If it were, those entries would carry a foreign sessionId; they don't. All six lines of the three groups carry the file's own id, and the file contains exactly one distinct sessionId across 425 lines — the filter never fires.

"Two-line groups get dropped" is also ruled out: 20 two-line thinking|tool_use groups survive in the same file.

I did not find the mechanism. Flagging it as open rather than guessing.

Suggested disposition

1. Amend the issue title/body to the reproducible numbers. As written it promises a 7× undercount that no longer exists on disk, which will send the next person looking for the wrong magnitude.
2. Cause 1 remains the fix that matters — union by step.id before write_cached, or minimally refuse to shrink Σ token_usage and warn. Everything downstream inherits the undercount, and it is the mechanism by which the original evidence was destroyed rather than merely miscounted.
3. The residual 3,511 deserves its own issue once the mechanism is known — it is a different failure from the shrink, and fixing cause 1 will not address it.

Method note: the transcript side of every number above is a run-grouped, field-wise-max walk of the raw JSONL (a group is a run of consecutive lines sharing message.id, per toolpath-convo/src/derive.rs:188; usage is a cumulative streaming snapshot, so summing lines within a group over-counts). Same rule the derive uses, applied independently.

[bdelanghe]

Correcting my comment above. The "3 groups / 3,511 output tokens missing per session" finding is wrong — it was an artefact of my measurement, not a defect in the derivation. There is no residual per-group loss. Apologies for the noise.

What I got wrong

I compared transcript and document by (group_id, output_tokens) pairs. Three ids came back as present in the transcript and absent from the document. They are not absent:

msg_011CeMqkikywmqz9PuXLNk94   doc carries: [1148]   transcript values: [1148]
msg_011CeMqoMX8Mk2wMbPJ6YJ39   doc carries: [1557]   transcript values: [1557]
msg_011CeMr4V9w4Zqq9gfGgRchp   doc carries: [806]    transcript values: [806]

The document has all three, with the right values. What differed was the count of usage carriers, because I was counting the transcript's groups per run of consecutive lines sharing message.id, and the document counts them per distinct message.id. Where an id appears in two non-adjacent runs, my side counted it twice.

Which of the two is correct

Per distinct message.id — the document. For every id in these sessions that spans more than one run, the usage tuple is identical across all its lines:

msg_011CeRb5bbvehAcPkVffsQ -> distinct usage tuples: 2/628/85469
msg_011CeRb6n31Zo1UTWk99DW -> distinct usage tuples: 2/813/87140
msg_011CeRbEio27C7i5zLwKP6 -> distinct usage tuples: 2/2014/98791

One tuple each (input/output/cache_read), not two. That is one API response whose content-block lines were split by an intervening entry — an Anthropic message.id names one response, so it cannot be two calls. Counting per run double-counts it.

So canonicalize_message_usage bounding on the message id is right, and the numbers in my table were the ones in error:

session	document (per id)	my per-run count	over-count
25095aad	102,354	105,865	+3,511
50357658	106,600	110,111	+3,511
5217d8e1	87,236	90,747	+3,511

(The delta is constant because the same three messages appear in all three transcripts via a replayed prefix.)

What still stands from that comment

• All four causes are still present on main @ b31f2c5 — cache.rs:57 truncate(true) with no union by step.id; no orphaned handling anywhere in toolpath-claude with is_bridge_entry still applied per-entry at lib.rs:157; derive.rs:49 still take(8); provider.rs:215 still one git show per Write. The three named test gaps are still absent.
• The 7× headline still does not reproduce. 25095aad's transcript is 427 lines with 72 message ids today, not 4,090 with 529, so the 784,105 figure is unverifiable from disk — which is what cause 1 predicts: the source shrank and the archive was replaced rather than merged.
• Cause 1 is still the fix that matters, and it is the one that destroyed the original evidence rather than merely miscounting it.

What does not stand is my suggestion to open a separate issue for the residual. There is no residual.