Skip to content

ClientTelemetry.tryInjectTracingContext only injects trace context for sampled spans, silently dropping it otherwise #400

Description

@antonmos

🐛 Current behavior

ClientTelemetry.tryInjectTracingContext only writes $traceId/$spanId into an event's user metadata when the current span is both valid and sampled:

private static List<EventData> tryInjectTracingContext(Span span, List<EventData> events) {
    if (!span.getSpanContext().isValid() || !span.getSpanContext().isSampled())
        return events;
    ...

https://github.com/kurrent-io/KurrentDB-Client-Java/blob/v1.2.1/src/main/java/io/kurrent/dbclient/ClientTelemetry.java#L28-L29

If the producing application's OTel SDK (or an upstream agent such as the Datadog Java agent) decides not to sample the current trace — which is a normal, expected outcome of head-based sampling, not an error condition — the appended event silently gets no trace context at all. Any consumer that later reads that event and tries to parent a span onto the producer (via tryExtractTracingContext) gets null and has no way to know a producer span ever existed. In an event-sourced system where the same event can be read by a subscription minutes, hours, or days after it was appended, this turns what should be a detached-but-linkable trace into a completely disconnected one, with no record that a producer ever existed.

Separately, tryExtractTracingContext always assumes the extracted context was sampled:

return SpanContext.createFromRemoteParent(traceId, spanId, TraceFlags.getSampled(),
        TraceState.getDefault());

https://github.com/kurrent-io/KurrentDB-Client-Java/blob/v1.2.1/src/main/java/io/kurrent/dbclient/ClientTelemetry.java#L78-L79

That happens to be self-consistent today only because injection is gated on isSampled() — but it means the sampling decision isn't actually round-tripped, it's just reconstructed from the fact that injection occurred at all.

🔍 Steps to reproduce

  1. Start a span with an unsampled OTel Sampler active (e.g. Sampler.alwaysOff(), or in practice: an agent like Datadog's whose head-based sampler drops the trace).
  2. Append an event to a stream while that span is current.
  3. Read the event's user metadata back (or via a subscription) — $traceId/$spanId are absent.

This is reproducible in a plain unit test against ClientTelemetry/EventDataBuilder with no live server required — the branch in question returns before any gRPC call is made.

💭 Expected behavior

Trace context should be written into event metadata regardless of the sampling decision, and the sampling decision itself should be carried alongside it (e.g. a $traceSampled boolean, or the two-hex-char W3C trace-flags form) rather than assumed on extraction. This mirrors how W3C Trace Context / OpenTelemetry Samplers are meant to be composed: the presence of a parent context and its sampled bit are two independent pieces of information, and only the consumer's own sampler should decide whether to keep the child span. Suppressing propagation entirely when the producer wasn't sampled removes information a downstream consumer might otherwise use to make its own (potentially different) sampling decision, and — as observed in production — silently breaks producer/consumer trace linkage for any event appended while the ambient trace was (correctly, by design) not sampled.

If gating injection on sampling is intentional (e.g. to avoid writing tracing metadata that will never be looked up), it would help at minimum to document that behavior clearly, since it's easy to build tracing instrumentation on top of this client that appears to work in testing (where everything is typically sampled) and then silently drops linkage for a fraction of production traffic.

Package version

kurrentdb-client 1.2.1 (same logic present since the telemetry feature was introduced; not new in this release)

KurrentDB Version

N/A — this is client-side logic that runs before any request reaches the server; not database-version-specific.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions