feat(tracing): correlate business spans with obs via dedicated wrapper span

[NiteshDhanpal]

What & why

Model the business ↔ observability correlation edge on the emit side, so a persisted business span can pivot to its Tempo/Datadog trace and back. No schema migration — the ids ride the existing operation_metadata JSONB (→ ClickHouse metadata_raw).

Two worlds today: business spans (trace_id = task id, stored in application_trace_span + CH) and obs traces (OTel/ddtrace in Tempo/Datadog) share no id. This adds the link.

Changes

• obs_ids.py — correlation keys standardized to obs_trace_id/obs_span_id (underscored → Postgres JSON-path addressable); removed the non-working dual mode (it needed an in-process ddtrace↔OTel bridge that can't exist; degrades to dd_only); obs_correlation() hardened to never raise.
• obs_span.py (new) — dedicated per-business-span wrapper obs span:
  • Opens a real span named for the step and makes it active, so obs_span_id is stable/meaningful (not an arbitrary innermost httpx span); nested instrumentation parents under it.
  • Backends: OTel (lgtm) / ddtrace (dd_only, only when a request trace is active — avoids orphan roots).
  • Reverse tag: stamps agentex.business_span_id / agentex.business_trace_id on the obs span → bidirectional pivot.
  • Error status propagated to the obs span (failed step is not a false green).
  • child_of nesting (ddtrace): ddtrace's start_span does not auto-parent — passes child_of=current_trace_context() so a turn's spans roll up into ONE trace instead of N roots.
• trace.py — wraps start_span/end_span (sync + async). Observability can never fail an app call (guarded; no-op when unconfigured); the two backends never interfere.
• temporal_task_service.py — opens a PRODUCER span around the ACP→Temporal dispatch (task/create / event/send) so the OTel interceptor injects context and the workflow/activity spans stay linked to the ingress trace (no detached roots). The span setup is fully fail-open — a broken provider or a raising sampler/SpanProcessor.on_start runs the dispatch untraced rather than failing task/create / event/send; the dispatch body's own exceptions still propagate.

Behavior (Turn 2 of the 3-turn example)

business step	before (obs_span_id)	with wrapper
get_state	rB	wB1
retrieve_docs	rB	wB2
chat_completion	rB	wB3
create_message	rB	wB4

Distinct, named obs_span_id per step; all under the one turn obs trace B.

✅ Live validation — infra-staging, lgtm/Tempo, end-to-end (sync + async paths)

Deployed to sgp-infra-staging in lgtm mode (Tempo) and validated end-to-end across both execution paths — a real agent turn renders as one connected obs trace with the business↔obs correlation edge populated in both directions.

• Async (Temporal) path — gateway → agent ACP (task/create / event/send) → Temporal workflow → activities → business spans: one connected trace (e.g. 12c6290e…, 347 spans), a single tree where every hop carries the ingress trace context — no detach at the ACP or Temporal boundaries. Business spans running inside activities anchor onto the interceptor's ambient span via the reverse tags (agentex.business_trace_id / agentex.business_span_id).
• Sync (in-process) path — FastACP(acp_type="sync"), rocket_mock_agent: each business step opens its own named wrapper obs span; obs_trace_id/obs_span_id persist to operation_metadata, distinct per step, all rolled up under the one per-turn trace.
• Bidirectional pivot confirmed: persisted business spans carry obs_trace_id/obs_span_id (obs → business, forward) and the obs spans carry agentex.business_* (business → obs, reverse), in Tempo.

Supersedes the earlier dd_only 52-span check — the child_of/wrapper behavior from that run still holds; this extends it to full lgtm/Tempo end-to-end coverage on both the sync and async paths.

Tests

Unit coverage for mode degrade + keys, both backends, non-interference, never-fails, reverse tag, error status, child_of nesting, and the Turn-2 example. Review-round additions (all green):

• test_adk_tracing_span_error.py — ADK span() CM records step failure (no false green) and never lets obs shadow the app exception.
• test_obs_span_fallback.py — invalid wrapper context → open_obs_span bails to None so the ambient obs_correlation() fallback still populates ids.
• test_obs_handle_registry.py — a raising processor can't crash the app path or leak; the handle registry is bounded (evict+close oldest).
• test_temporal_obs_backend.py — Temporal path tags/reads the OTel interceptor span even in dd_only (mode-independent), with ddtrace fallback.

Not in this PR (follow-ups)

• business_trace_id on every sub-span via OTel baggage — split out to its own stack (feat/obs-baggage-business-trace).
• The sgp-obs-tracing-middleware library still carries dual/bridge.py — separate cleanup.

🤖 Generated with Claude Code

Greptile Summary

The PR adds bidirectional correlation between business spans and observability traces.

• Creates dedicated OTel or Datadog wrapper spans for synchronous business-span lifecycles.
• Correlates Temporal business spans with ambient interceptor spans and propagates ACP dispatch context.
• Persists standardized observability IDs in business-span metadata and records reverse business identifiers on observability spans.
• Makes tracing processor, queue, and wrapper cleanup failures fail-open while bounding the live-handle registry.
• Adds focused coverage for correlation, error propagation, fallback behavior, Temporal integration, and handle cleanup.

Confidence Score: 5/5

The PR appears safe to merge.

The previously reported processor-failure leak is resolved because synchronous processor exceptions and asynchronous enqueue failures are contained after handle registration, while the normal end path removes and closes the wrapper handle; no blocking failure remains.

Important Files Changed

Filename	Overview
src/agentex/lib/core/tracing/trace.py	Integrates wrapper-span lifecycle management into synchronous and asynchronous business spans, including bounded cleanup and fail-open processor handling.
src/agentex/lib/core/tracing/obs_span.py	Adds backend-specific wrapper creation, reverse tagging, error propagation, ambient Temporal tagging, and guarded cleanup.
src/agentex/lib/core/tracing/obs_ids.py	Standardizes persisted correlation keys, removes dual-mode selection, and makes ID extraction fail-open.
src/agentex/lib/core/temporal/services/temporal_task_service.py	Wraps task creation and event signaling in producer spans so Temporal interceptors can propagate active context.
src/agentex/lib/adk/_modules/tracing.py	Records context-manager exceptions on business spans before ending them so correlated observability spans receive error status.
tests/test_obs_handle_registry.py	Verifies processor failures cannot escape or leak handles and confirms bounded registry eviction behavior.
tests/test_obs_span_fallback.py	Verifies invalid wrapper contexts retain ambient correlation fallback.
tests/test_temporal_obs_backend.py	Verifies mode-independent Temporal OTel correlation and ddtrace fallback behavior.

Sequence Diagram

sequenceDiagram
    participant App
    participant Business as Business Span
    participant Wrapper as Obs Wrapper
    participant Backend as Tempo/Datadog
    App->>Business: start_span(step)
    Business->>Wrapper: open named active span
    Wrapper-->>Business: obs_trace_id + obs_span_id
    Business->>Business: persist IDs in metadata
    Wrapper->>Wrapper: tag business_trace_id + business_span_id
    App->>Business: end_span(error?)
    Business->>Wrapper: close and propagate error status
    Wrapper->>Backend: export correlated span

Loading

_{Reviews (7): Last reviewed commit: "test(tracing): update obs_span invalid-c..." | Re-trigger Greptile}