Summary
Implement the shared OTel topology defined in aws/aws-durable-execution-conformance-tests#99.
The durable backend normally supplies a stable remote server span through _X_AMZN_TRACE_ID for every reinvocation. Use a valid Root and Parent as the authoritative execution trace regardless of whether Sampled is present. Create a deterministic synthetic execution root only when a valid remote parent cannot be constructed.
Required changes
- Extend
ExtractedContext and XRayContextExtractor to parse Root, Parent, and Sampled independently; only Sampled=0 and Sampled=1 are authoritative upstream decisions.
- Create
Workflow with the remote server context as parent whenever Root and Parent are valid, inheriting its trace ID while retaining the deterministic Workflow span ID.
- For
Invocation, prefer the current ambient span only when its trace ID matches the remote context; otherwise use the remote server span.
- Do not switch to a synthetic root merely because
Sampled is absent or unusable.
- When a valid remote
Parent cannot be constructed, ignore unrelated ambient context and parent both Workflow and all Invocation spans to the synthetic execution root.
- Give the synthetic execution root a deterministic span ID stable across reinvocations and distinct from Workflow and operation span ID namespaces.
- Restore continuation and replay links to the initial logical operation span using the canonical trace ID and
generateSpanIdForOperation(durableExecutionArn, operationId).
- Retain the Workflow correlation link where required by the shared contract.
- Remove remaining hard-coded
TraceFlags.getSampled() values from extracted and reconstructed durable contexts.
Header resolution and sampling
_X_AMZN_TRACE_ID state |
Canonical trace ID |
Common execution ancestor |
Sampling |
Valid Root, Parent, Sampled=1 |
Reuse Root |
Remote Parent |
Preserve sampled |
Valid Root, Parent, Sampled=0 |
Reuse Root |
Remote Parent |
Preserve not-sampled |
Valid Root, Parent, no valid Sampled |
Reuse Root |
Remote Parent |
Leave sampled trace flag unset; configured sampler behavior applies |
Valid Root, missing or invalid Parent, Sampled=1 or Sampled=0 |
Reuse Root |
Synthetic execution root |
Preserve explicit decision |
Valid Root, missing or invalid Parent, no valid Sampled |
Reuse Root |
Synthetic execution root |
Configured root sampler decides |
Missing or invalid Root |
Derive from execution ARN and stable execution start time |
Synthetic execution root |
Configured root sampler decides |
An absent or unusable Sampled value must not be treated as an explicit Sampled=0, although its OTel span context has the sampled bit unset. ParentBased therefore treats the remote parent as not sampled, while a directly configured non-parent-based TraceIdRatioBased sampler can decide from the canonical trace ID. Trace-ID-ratio decisions remain stable across reinvocations because the canonical trace ID is stable.
Span hierarchy
Valid Root and Parent, with same-trace ambient Lambda spans when available:
Remote backend server span (`Root` / `Parent`)
├── Workflow
├── Ambient Lambda span 1
│ └── Invocation 1
├── Ambient Lambda span 2
│ └── Invocation 2
└── Invocation N [direct child when no valid same-trace ambient span exists]
Missing or invalid remote Parent fallback:
Synthetic execution root
├── Workflow
├── Invocation 1
├── Invocation 2
└── Invocation N
Plugin-specific operation placement:
ExecutionOtelPlugin
Workflow
└── Operation span
└── link -> current Invocation
InvocationOtelPlugin
Invocation
└── Operation segment
├── link -> Workflow
└── continuation/replay link -> initial logical operation span
Links are correlations and do not replace the parent-child edges shown above.
Tests
Add focused coverage for both OTel plugins:
- every header-resolution table row, including
Sampled=0, Sampled=1, and absent Sampled;
- missing-
Sampled behavior with ParentBased and direct trace-ID-ratio samplers while retaining the real remote parent;
- configured root-sampler behavior for missing-parent cases;
- both remote-parent and synthetic-root span hierarchies;
- deterministic synthetic-root identity and common Workflow/Invocation parentage across reinvocations;
- ambient parent accepted only for the same remote trace;
- unrelated ambient context rejected in normal and fallback modes;
- continuation and replay links resolving to the original deterministic operation span;
- multi-invocation wait/resume, retries, and chained executions;
- ADOT global-provider and explicit-provider-builder modes.
Update the module README and examples that currently describe two trace domains or a parentless Workflow span.
Related
Summary
Implement the shared OTel topology defined in aws/aws-durable-execution-conformance-tests#99.
The durable backend normally supplies a stable remote server span through
_X_AMZN_TRACE_IDfor every reinvocation. Use a validRootandParentas the authoritative execution trace regardless of whetherSampledis present. Create a deterministic synthetic execution root only when a valid remote parent cannot be constructed.Required changes
ExtractedContextandXRayContextExtractorto parseRoot,Parent, andSampledindependently; onlySampled=0andSampled=1are authoritative upstream decisions.Workflowwith the remote server context as parent wheneverRootandParentare valid, inheriting its trace ID while retaining the deterministic Workflow span ID.Invocation, prefer the current ambient span only when its trace ID matches the remote context; otherwise use the remote server span.Sampledis absent or unusable.Parentcannot be constructed, ignore unrelated ambient context and parent both Workflow and all Invocation spans to the synthetic execution root.generateSpanIdForOperation(durableExecutionArn, operationId).TraceFlags.getSampled()values from extracted and reconstructed durable contexts.Header resolution and sampling
_X_AMZN_TRACE_IDstateRoot,Parent,Sampled=1RootParentRoot,Parent,Sampled=0RootParentRoot,Parent, no validSampledRootParentRoot, missing or invalidParent,Sampled=1orSampled=0RootRoot, missing or invalidParent, no validSampledRootRootAn absent or unusable
Sampledvalue must not be treated as an explicitSampled=0, although its OTel span context has the sampled bit unset.ParentBasedtherefore treats the remote parent as not sampled, while a directly configured non-parent-basedTraceIdRatioBasedsampler can decide from the canonical trace ID. Trace-ID-ratio decisions remain stable across reinvocations because the canonical trace ID is stable.Span hierarchy
Valid
RootandParent, with same-trace ambient Lambda spans when available:Missing or invalid remote
Parentfallback:Plugin-specific operation placement:
Links are correlations and do not replace the parent-child edges shown above.
Tests
Add focused coverage for both OTel plugins:
Sampled=0,Sampled=1, and absentSampled;Sampledbehavior withParentBasedand direct trace-ID-ratio samplers while retaining the real remote parent;Update the module README and examples that currently describe two trace domains or a parentless Workflow span.
Related