[bot] Google GenAI: streaming responses drop `url_context_metadata` that non-streaming responses preserve

[braintrust-bot]

<!-- provider-gap-audit: google-genai-streaming-url-context-metadata -->

Summary

When the Gemini url_context tool is used with generate_content_stream() / agenerate_content_stream(), the per-URL retrieval metadata (candidate.url_context_metadata) that Google's SDK returns is silently dropped from the Braintrust span output. The equivalent metadata for the google_search tool (candidate.grounding_metadata) is preserved in the exact same code path — so this is a fidelity gap between two structurally-analogous tool result types within the same function, not a "feature never built" gap.

Non-streaming generate_content() calls do not have this problem: the raw GenerateContentResponse object (including url_context_metadata) is logged as-is, so nothing is lost there.

What is missing

_aggregate_generate_content_chunks() in py/src/braintrust/integrations/google_genai/tracing.py (used by both the sync and async streaming wrappers) manually reconstructs a candidate_dict from the accumulated chunks, copying over only an explicit allowlist of candidate fields:

candidate_dict = {"content": {"parts": parts, "role": "model"}}

if hasattr(candidate, "finish_reason"):
    candidate_dict["finish_reason"] = candidate.finish_reason
if hasattr(candidate, "safety_ratings"):
    candidate_dict["safety_ratings"] = candidate.safety_ratings
if hasattr(candidate, "grounding_metadata") and candidate.grounding_metadata:
    candidate_dict["grounding_metadata"] = candidate.grounding_metadata

(py/src/braintrust/integrations/google_genai/tracing.py:695-709)

candidate.url_context_metadata — the field Google's own docs say to inspect to see "which URLs the model retrieved" when the url_context tool is enabled — is never copied into candidate_dict, so it never reaches the logged span output for streaming calls. Any user who calls client.models.generate_content_stream(..., config=GenerateContentConfig(tools=[{"url_context": {}}])) gets a span with no record of which URLs were actually fetched, even though the same call via generate_content() (non-streaming) would show it.

This is the same class of field (candidate.<x>_metadata describing what a built-in tool did) as grounding_metadata, which is explicitly captured here and has dedicated test coverage (test_google_search_grounding / test_google_search_grounding_async in test_google_genai.py). There is no equivalent test for url_context, and a full-file grep for url_context or code_execution in test_google_genai.py returns zero matches — confirming there is no regression coverage that would have caught this gap.

Note: the separate _TOOL_CALL_TYPES/_TOOL_RESULT_TYPES constants and interaction-tool-span logic elsewhere in the same file (tracing.py:54-69, :877-969) do already generically recognize url_context_call/url_context_result and code_execution_call/code_execution_result — that mechanism belongs to the newer content-item/"interactions" API surface and is unrelated to the classic generate_content_stream() candidate-based aggregation described above, which is the specific path where the metadata is lost.

Braintrust docs status

not_found — https://www.braintrust.dev/docs/integrations/ai-providers/google-genai (and the general https://www.braintrust.dev/docs/guides/tracing) do not document url_context tool support or grounding/citation-style metadata capture at all, streaming or otherwise.

Upstream sources

• Google AI for Developers, URL context tool guide (shows response.candidates[0].url_context_metadata as the way to verify which URLs were retrieved): https://ai.google.dev/gemini-api/docs/generate-content/url-context
• Vertex AI GenerateContentResponse/Candidate reference (documents url_context_metadata as a candidate-level field, analogous to grounding_metadata): https://docs.cloud.google.com/vertex-ai/generative-ai/docs/reference/rest/v1/GenerateContentResponse
• google-genai Python SDK (google.genai.types), which defines UrlContextMetadata / UrlMetadata on the candidate object

Local repo files inspected

• py/src/braintrust/integrations/google_genai/tracing.py:
  • _aggregate_generate_content_chunks() (~lines 641-722) — builds candidate_dict for streaming span output; copies finish_reason, safety_ratings, grounding_metadata but not url_context_metadata
  • _gc_process_result() (~lines 573-581) — non-streaming path; returns the raw GenerateContentResponse, so no loss there
  • _TOOL_CALL_TYPES / _TOOL_RESULT_TYPES (~lines 54-69) and the interaction-tool-span logic (~lines 877-969) — confirmed this is a separate code path (content-item/interactions API) unrelated to the candidate-based streaming aggregation gap above
• py/src/braintrust/integrations/google_genai/test_google_genai.py:
  • test_google_search_grounding / test_google_search_grounding_async (~lines 1451, 1557) and _assert_grounding_metadata (~line 1411) — dedicated grounding-metadata test exists for google_search only
  • Full-file grep for url_context and code_execution — zero matches, confirming no test coverage for either tool type