Propagate LLM Observability context across service boundaries

[ncybul]

What Does This Do

Carries LLM Observability context — ml_app, session_id, and agent attribution — across process boundaries, so an LLM trace stays whole when work crosses a queue or a service call.

The driving case is SQS: an agent dispatches work onto a queue, and the worker that picks it up currently starts with no session and no agent attribution, fragmenting what is logically one trace.

Motivation

LLMObs context has been in-process only. LLMObsContext propagates session_id and agent attribution to descendant spans on the same thread, but nothing reaches the wire — so every service boundary splits the trace.

dd-trace-py already does this automatically: LLMObs subscribes to the generic http.span_inject hook that HTTPPropagator.inject fires on every outbound request, and writes _dd.p.llmobs_* onto the span context there. Any auto-instrumented call carries LLMObs context with no application code. This brings Java to the same behaviour, using the same tag names, so a mixed-language pipeline joins up.

Approach

Rather than teaching each integration about LLMObs, register an LLMObsContextPropagator as a propagation concern (AgentPropagation.LLMOBS_CONCERN).

It contributes no headers of its own. It stages the _dd.p.llmobs_* tags onto the span context being injected, and the tracing propagator then serializes them like any other propagation tag. Composite injection runs in reverse priority order, so the concern sorts after TRACING_CONCERN (priority 115 vs 100) in order to inject before it. This is the same shape as the existing DSM_CONCERN, which likewise stashes state on the span context rather than writing headers.

Consequences:

• Every boundary automatic instrumentation already covers is handled at once — SQS, HTTP, gRPC, Kafka — with no integration-specific code.
• Values are resolved from the ambient LLMObsContext at injection time, not written once when a span starts. The innermost active LLMObs span wins, and leaving a scope stops contributing tags, with no save/restore bookkeeping.
• Injection is gated on trace-id consistency, matching the gate DDLLMObsSpan already applies to in-process inheritance.

Why SQS needs no code of its own

SqsInterceptor.modifyRequest already injects through defaultPropagator() into the _datadog message attribute, and on the receive side TracingIterator extracts it (Config.isSqsPropagationEnabled() defaults to true) and activates the consume span while the consumer's per-message code runs (TracingIterator.java:117-124). So once the concern is registered, both directions work: the producer's tags ride the existing _datadog attribute, and the worker's LLMObs spans inherit from the activated consume span.

Receive side

DDLLMObsSpan now reads session_id and agent attribution off the propagated context whenever no same-trace in-process parent contributed them. Note this deliberately covers the trace-mismatch case as well as the no-parent case: a stale LLMObsContext leaked from an unrelated trace must not suppress attribution that legitimately arrived over the wire — which is exactly what an SQS worker hits when a previous message's scope leaks on the same thread.

ml_app moves into LLMObsContext (it previously lived only as a span tag), so the propagator can read the innermost active span's value without holding a reference to the span.

Behaviour change

This is on by default whenever LLM Observability is enabled: outbound requests made while an LLMObs span is active now carry a few extra tag bytes in x-datadog-tags / tracestate. That matches dd-trace-py. Happy to put it behind a config flag if reviewers would prefer an opt-in.

Relationship to #12402

#12402 adds a manual LLMObs.injectDistributedHeaders / activateDistributedHeaders API for carriers no instrumentation reaches. That API is not in this PR — this is the automatic path only, which is what the SQS use case actually needs. #12402 will rebase on top of this and keep only the public manual API.

Testing

LLMObsContextPropagatorTest — five tests, none of which call any LLMObs propagation API:

• Injecting the active span the way auto-instrumentation does carries ml_app, session_id, pagent_span_id, pagent_name

• Nothing is added when no LLMObs span is active

• Tags stop being contributed once the LLMObs scope closes

• Producer → worker round trip: a worker inherits session and agent attribution across the boundary with zero application plumbing, using a Map carrier — the shape the SQS _datadog attribute reduces to at the propagator boundary

• A worker with no upstream LLMObs context inherits nothing

• :dd-java-agent:agent-llmobs:test — green

• :internal-api:test — green

• :dd-trace-core:test --tests "datadog.trace.core.propagation.*" — green

Note: PendingTraceBufferTest and TracerConnectionReliabilityTest fail in the full dd-trace-core suite. These were confirmed earlier to fail identically on unmodified master and are unrelated to this change.

Follow-ups

• An integration test in aws-java-sqs-2.0 exercising this against the real AWS SDK, alongside SqsClientTest.java. That module does not currently have agent-llmobs on its test classpath, so it is a non-trivial addition and is left out here.
• Round-trip unit tests for the new tags in DatadogPTagsCodec / W3CPTagsCodec.

[datadog-prod-us1-6]

🎯 Code Coverage (details)
• Patch Coverage: 34.32%
• Overall Coverage: 58.59% (-0.45%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a66d66d | Docs | View more details | Give us feedback!}

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.03 s	13.97 s	[-0.4%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent	12.94 s	12.98 s	[-1.0%; +0.3%] (no difference)
startup:petclinic:appsec:Agent	16.98 s	16.85 s	[-0.2%; +1.8%] (no difference)
startup:petclinic:iast:Agent	16.86 s	16.95 s	[-1.3%; +0.3%] (no difference)
startup:petclinic:profiling:Agent	16.72 s	16.96 s	[-2.5%; -0.2%] (maybe better)
startup:petclinic:sca:Agent	16.94 s	16.61 s	[+0.9%; +3.1%] (maybe worse)
startup:petclinic:tracing:Agent	16.05 s	16.12 s	[-1.5%; +0.5%] (no difference)

Commit: a66d66d7 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.