fix(opentelemetry): resolve lazy sampling + distributed tracing bug [backport 2.9] #10028
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport 47aad1c from #9974 to 2.9.
Description
With ddtrace v2.8.0 sampling decisions are no longer made when a root span is created. Instead sampling decisions are lazily evaluated when trace information is expected to leave a process. The ddtrace library sets a sampling decision on root spans when one of the following conditions are met:
Spans generated using OpenTelemetry API (via ddtrace TracerProvider) do not use the ddtrace HttpPropagator when propagating distributed traces. This leaves an edge case where the opentelemetry-api can propagate a distributed trace before a sampling decision is made. Since the default state of an opentelemetry span is unsampled, downstream services could receive a traceflag of
00
and drop spans that should have been kept. This could result in missing spans/incomplete traces in the Datadog UI.Fix
This PR ensures that sampling decisions are ALWAYS made before a SpanContext is extracted from an OpenTelemetry span. Since
Span.get_span_context()
is the only mechanism to extract/propagate tracing information (ex: sampling decision, trace_id, span_id, etc.) from an OpenTelemetry Span, making a sampling decision here will ensure the OpenTelemetry API never propagates an undefined sampling decision. IfSpan.get_span_context()
is never invoked, then OpenTelemetry spans will continue to be lazily sampled on serialization (just like Datadog spans).TODO
There are many cases where trace information from a Datadog span can escape a process before a sampling decision is made (ex: via threads, spawned processes, manual context propagation). For these scenarios we ask users to manually sample spans (via these docs). Ideally user's should NOT need to kno the internal workings of tracer sampling and they should not be required to call
tracer.sample(span)
in their applications to resolve missing span issues. Sampling decisions should ALWAYS be made when tracing internals accessSpan.context.sampling_priority
for the first time.With this approach an invalid/undefined sampling priority is never returned. The Datadog Span Context will always return a consistent sampling decision.
cc: @brettlangdon, @zacharycmontoya, @ZStriker19
Risk
This change makes "lazy sampling" less "lazy" (for the OpenTelemetry API). Span tags and resource names set after
Span.get_span_context()
is called will NOT be used to make a sampling decision.Checklist
Reviewer Checklist