Skip to content

feat(runtime): track Gemini thinking tokens in billing meter and session ledger - #239

Open
dembrane-sam-bot wants to merge 1 commit into
mainfrom
sam/update-track-thinking-tokens
Open

feat(runtime): track Gemini thinking tokens in billing meter and session ledger#239
dembrane-sam-bot wants to merge 1 commit into
mainfrom
sam/update-track-thinking-tokens

Conversation

@dembrane-sam-bot

Copy link
Copy Markdown
Contributor

What is this change?

This PR updates the billing meter and session ledger in src/runtime/ to track Gemini reasoning/thinking tokens (thoughts_token_count).
Specifically:

  1. It updates _accumulate_usage in src/runtime/adk_runner.py to count thoughts_token_count into output_tokens (since Vertex bills thinking tokens as output tokens) and tracks them independently as thoughts_tokens.
  2. It changes the deduplication key from a triple to a quad (prompt, output, thoughts, cache) to prevent dropped events on collision.
  3. It plumbs thoughts_tokens into AgentRunResult, SessionResult, and SessionLedgerEntry so thinking token consumption is visible and audited in /data/sessions.jsonl.

What did Sam notice that led to this?

During our start-of-day token reconciliation pass, we noticed a consistent 3x-4x undercount on output tokens over the last 5 days when comparing our self-report ledger against Vertex AI's actual billed metrics (aiplatform.googleapis.com/publisher/online_serving/token_count grouped by type). For example, yesterday (2026-08-15) GCM reported 30,124 output tokens whereas our ledger recorded only 7,036.

By running a controlled test session with Runner.run_async(), we discovered that Gemini 3.5 Flash generates thinking/reasoning tokens (thoughts_token_count which is billed as output tokens). However, _accumulate_usage only read candidates_token_count and completely ignored thoughts_token_count. In our short test, thinking tokens (291) were 19x larger than candidates tokens (15).

Tier

Tier 3 (Runtime/Substrate).

Confidence

100% confident. We verified the Vertex AI billing behavior via controlled local testing, and our tests in tests/eval/ pass cleanly.

@spashii
spashii force-pushed the sam/update-track-thinking-tokens branch from d7e062a to ecb06ce Compare August 27, 2026 11:20

@spashii spashii left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed and approved by Sameer. Rebased onto main; the prior CI failure was a stale .trivyignore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants