Skip to content

otel: parent Workflow to backend context and restore operation links #644

Description

@zhongkechen

Summary

Implement the shared OTel topology defined in aws/aws-durable-execution-conformance-tests#99.

The durable backend normally supplies a stable remote server span through _X_AMZN_TRACE_ID for every reinvocation. Use a valid Root and Parent as the authoritative execution trace regardless of whether Sampled is present. Create a deterministic synthetic execution root only when a valid remote parent cannot be constructed.

Required changes

  • Extend ExtractedContext and XRayContextExtractor to parse Root, Parent, and Sampled independently; only Sampled=0 and Sampled=1 are authoritative upstream decisions.
  • Create Workflow with the remote server context as parent whenever Root and Parent are valid, inheriting its trace ID while retaining the deterministic Workflow span ID.
  • For Invocation, prefer the current ambient span only when its trace ID matches the remote context; otherwise use the remote server span.
  • Do not switch to a synthetic root merely because Sampled is absent or unusable.
  • When a valid remote Parent cannot be constructed, ignore unrelated ambient context and parent both Workflow and all Invocation spans to the synthetic execution root.
  • Give the synthetic execution root a deterministic span ID stable across reinvocations and distinct from Workflow and operation span ID namespaces.
  • Restore continuation and replay links to the initial logical operation span using the canonical trace ID and generateSpanIdForOperation(durableExecutionArn, operationId).
  • Retain the Workflow correlation link where required by the shared contract.
  • Remove remaining hard-coded TraceFlags.getSampled() values from extracted and reconstructed durable contexts.

Header resolution and sampling

_X_AMZN_TRACE_ID state Canonical trace ID Common execution ancestor Sampling
Valid Root, Parent, Sampled=1 Reuse Root Remote Parent Preserve sampled
Valid Root, Parent, Sampled=0 Reuse Root Remote Parent Preserve not-sampled
Valid Root, Parent, no valid Sampled Reuse Root Remote Parent Leave sampled trace flag unset; configured sampler behavior applies
Valid Root, missing or invalid Parent, Sampled=1 or Sampled=0 Reuse Root Synthetic execution root Preserve explicit decision
Valid Root, missing or invalid Parent, no valid Sampled Reuse Root Synthetic execution root Configured root sampler decides
Missing or invalid Root Derive from execution ARN and stable execution start time Synthetic execution root Configured root sampler decides

An absent or unusable Sampled value must not be treated as an explicit Sampled=0, although its OTel span context has the sampled bit unset. ParentBased therefore treats the remote parent as not sampled, while a directly configured non-parent-based TraceIdRatioBased sampler can decide from the canonical trace ID. Trace-ID-ratio decisions remain stable across reinvocations because the canonical trace ID is stable.

Span hierarchy

Valid Root and Parent, with same-trace ambient Lambda spans when available:

Remote backend server span (`Root` / `Parent`)
├── Workflow
├── Ambient Lambda span 1
│   └── Invocation 1
├── Ambient Lambda span 2
│   └── Invocation 2
└── Invocation N                 [direct child when no valid same-trace ambient span exists]

Missing or invalid remote Parent fallback:

Synthetic execution root
├── Workflow
├── Invocation 1
├── Invocation 2
└── Invocation N

Plugin-specific operation placement:

ExecutionOtelPlugin
Workflow
└── Operation span
    └── link -> current Invocation

InvocationOtelPlugin
Invocation
└── Operation segment
    ├── link -> Workflow
    └── continuation/replay link -> initial logical operation span

Links are correlations and do not replace the parent-child edges shown above.

Tests

Add focused coverage for both OTel plugins:

  • every header-resolution table row, including Sampled=0, Sampled=1, and absent Sampled;
  • missing-Sampled behavior with ParentBased and direct trace-ID-ratio samplers while retaining the real remote parent;
  • configured root-sampler behavior for missing-parent cases;
  • both remote-parent and synthetic-root span hierarchies;
  • deterministic synthetic-root identity and common Workflow/Invocation parentage across reinvocations;
  • ambient parent accepted only for the same remote trace;
  • unrelated ambient context rejected in normal and fallback modes;
  • continuation and replay links resolving to the original deterministic operation span;
  • multi-invocation wait/resume, retries, and chained executions;
  • ADOT global-provider and explicit-provider-builder modes.

Update the module README and examples that currently describe two trace domains or a parentless Workflow span.

Related

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingdocumentationImprovements or additions to documentationparityProvides parity with other language implementations of the SDKpkg:otelModule: otel-plugin

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions