---

Abstract #

When one user request fans out across agents, tool servers, queues and callbacks, one missing trace header splits its evidence into traces that no backend can rejoin by trace ID. We built a two-agent, six-tool chain instrumented with the OpenTelemetry Python SDK 1.45.1 over real localhost transports (a thread-pool hand-off, agent-to-agent JSON-RPC over HTTP, Model Context Protocol (MCP) calls with context in params._meta, a queue and a webhook), removed context at chosen and random hops, measured bytes and latency per carrier, simulated 20,000 requests under head and tail sampling, and scored 13 specifications and framework pages on propagation. First, a missing _meta field left 25.0% of a request’s 48 spans in the root trace and split it into 7 traces; at 95% per-hop coverage across 21 hops only 39.0% of requests stayed fully connected. Second, full context (traceparent, tracestate, baggage) added 232 to 248 bytes per message and 54.8 to 69.5 microseconds per localhost round trip. Third, at a 5% base rate, 10-second median queue waits and the collector’s default 30-second decision wait, tail sampling kept 22.30% of complete failing traces against 4.12% for parent-based head sampling, and a decision cache raised that to 80.45%.

1. Introduction #

The previous article in this series, Explainability-as-a-Service[1]^[2], priced the platforms that store traces; this one asks whether the traces they store are whole. The series opened with the argument that OpenTelemetry alone does not capture what AI systems need[2]. Multi-agent systems add a more basic problem: the trace itself falls apart as a request crosses agents, tool servers, queues and callbacks.

That matters because debugging a multi-agent failure means finding the agent and the step that caused it. The MAST study annotated more than 1,600 traces from 7 frameworks and found 14 failure modes in 3 categories, one of which is inter-agent misalignment[3]^[3]. On the Who&When benchmark, the best automated method identified the responsible agent in 53.5% of cases and the decisive step in only 14.2%[4]^[4]. Observability is part of the reason: TraceElephant reports that full execution traces improve attribution accuracy by up to 76% over traces that keep only agent outputs[5]^[5]. A trace that comes apart at a hop is worse than partial: it becomes several unrelated-looking traces.

The wire formats exist. W3C Trace Context defines the traceparent and tracestate headers[6]^[6], MCP’s SEP-414 (status Final) puts those keys in _meta[7]^[7], and OpenTelemetry’s MCP conventions tell instrumentations to inject context there[8]^[8]. What is missing is a measurement of what each gap costs, what carrying context costs, and how sampling interacts with asynchronous hops.

RQ1: When trace context is missing at one hop type of a multi-agent tool chain, what share of a request’s spans stays in the root trace, and how does end-to-end continuity scale with per-hop propagation coverage? RQ2: What does carrying W3C trace context, with tracestate and baggage, cost per hop in bytes and latency on HTTP, MCP JSON-RPC and queue carriers? RQ3: At a given base sampling rate, what share of complete failing multi-hop traces do head sampling and collector tail sampling keep when hops are asynchronous?

2. Existing Approaches (2025–2026 state of the art) #

graph TD
    A[Cross-agent traceability] --> B[In-band context propagation]
    A --> C[Out-of-band correlation]
    A --> D[Reconstruction after the fact]
    A --> E[Sampling and retention]
    B --> B1[W3C headers on HTTP]
    B --> B2[MCP params meta]
    B --> B3[Queue headers and span links]
    C --> C1[Conversation and task IDs]
    C --> C2[Provenance graphs]
    D --> D1[eBPF boundary tracing]
    D --> D2[Failure attribution models]
    E --> E1[Head sampling]
    E --> E2[Tail sampling at collector]

2.1 In-band propagation: W3C Trace Context on every carrier #

In-band propagation starts from W3C Trace Context. A traceparent value carries a 16-byte trace ID, an 8-byte parent ID and trace flags; Level 2, still a Candidate Recommendation Draft dated 28 March 2024, adds a random-trace-id flag which, when set, requires that at least the right-most 7 bytes of the trace ID be random[6]^[6]. Vendors “SHOULD propagate at least 512 characters” of a combined tracestate, which may hold at most 32 list members[6]^[6].

For MCP, SEP-414 states that when trace context travels in _meta, the keys traceparent, tracestate and baggage follow the W3C value formats, as an exception to MCP’s rule that _meta keys carry a DNS prefix[7]^[7]. The OpenTelemetry MCP conventions add the reason: “HTTP trace context propagation only covers the HTTP request, but not the individual messages client and server exchange within the request/response streams”, and one MCP request can span several HTTP requests on retry while one streamable HTTP request can serve several MCP messages[8]^[8]. Context therefore belongs in the message, with the transport context recorded as a span link.

The Agent2Agent (A2A) protocol takes a different route. Its v1.0 specification lists tracing among its enterprise-ready principles and defines a contextId that groups related tasks and messages, but its data model has no trace field[9]^[9]. Its enterprise guide says clients and servers “should participate in distributed tracing systems”, for example with W3C Trace Context headers[10]^[10]. That covers synchronous HTTP; the specification’s push notifications, which deliver results to a client webhook later, get no comparable guidance.

2.2 Asynchronous hops: span links instead of parents #

Queues strain the parent-child model: a message can be consumed in batches, long after the producer finished. The OpenTelemetry messaging conventions propagate a “message creation context” with the message and use span links as the default way to correlate producers and consumers, because links are “the only consistent trace structure that can be guaranteed” across messaging systems[11]^[11]. In the convention’s example of push delivery over HTTP, the default makes the consumer’s process span a child of the HTTP server span, with a link to the creation context. A backend that follows only parent IDs will therefore show the consumer side as a separate trace.

2.3 Framework-level propagation and its boundaries #

Microsoft Agent Framework injects traceparent and tracestate into params.meta of tools/call requests when an OpenTelemetry span is active, but only for MCP sessions the agent process opens itself. For hosted MCP tools the provider’s runtime issues the call, so the framework “has no opportunity to inject trace context into those requests”[12]^[12]. The OpenAI Agents SDK has its own trace model, with IDs in the format trace<32_alphanumeric> and an optional group_id that links traces from one conversation[13]^[13]. Its page describes no cross-process carrier, and such an ID is a valid W3C trace ID only if it happens to be lowercase hexadecimal.

2.4 Correlation after the fact: attribution and provenance #

When propagation fails, research recovers structure from logs. AgenTracer reports that reasoning LLMs generally reach below 10% accuracy on failure attribution and trains an 8-billion-parameter tracer through counterfactual replay and fault injection[14]^[14]. TRAIL’s 148 human-annotated traces left the best long-context model at 11%[15]^[15]. MP-Bench argues that many failures admit several plausible attributions[16]^[16]. FAMAS ranks agent actions by spectrum analysis over repeated runs[17]^[17], and GraphTracer builds information dependency graphs because temporal order alone does not show which agent built on which output[18]^[18]. All of them reconstruct dependencies that propagated context would record directly.

Provenance work makes that structure explicit. A 2026 survey of execution provenance found no benchmark in its catalogue that evaluates the complete provenance chain and little reporting of tamper-evident traces[19]^[19]. PROV-AGENT extends W3C PROV and uses MCP to link agent prompts, responses and decisions into end-to-end workflow provenance across edge, cloud and HPC[20]^[20]. Instrumentation-free tracing is the alternative: AgentSight (PACMI 2025) correlates intercepted LLM traffic with kernel events across process boundaries using eBPF at under 3% overhead[21]^[21], and CrossTrace embeds span identifiers in TCP headers via eBPF and correlates spans with over 90% accuracy[22]^[22].

2.5 Sampling: deciding which traces to keep #

Tracezip summarises the trade-off: head sampling “indiscriminately selects requests” and may miss critical events, while tail sampling must first collect everything[23]^[23]. Span-level sampling keeps every trace in a thinner form: Autoscope cut trace size by 81.2% while keeping 98.1% of faulty spans[24]^[24], and RADAR, a reinforcement-learning sampler on OpenTelemetry, reduced bandwidth by 97.4% against full collection while retaining about 85.6% of rare trace patterns[25]^[25]. None uses agent workloads, whose hops can wait for hours.

3. Method and Data #

All code, data and outputs are at stabilarity/hub/research/cross-agent-traceability-propagating-observability-context-through-multi-agent-tool-chains. The script runs offline in about 70 seconds and writes every number below to results.json. Hardware and software: an AMD EPYC virtual machine at 2.0 GHz with 6 vCPUs, Linux 6.8, Python 3.12.3, opentelemetry-sdk 1.45.1.

graph LR
    U[User request] --> O[Orchestrator root span]
    O -->|in-process thread pool| P[Planner]
    P -->|A2A JSON-RPC over HTTP| R1[Research agent 1]
    P -->|A2A JSON-RPC over HTTP| R2[Research agent 2]
    R1 -->|MCP tools call x3| T[Tool server]
    R2 -->|MCP tools call x3| T
    T -->|queue headers| W[Worker and LLM step]
    W -->|webhook push| O

Continuity experiment. Each request follows the topology above over real transports: a thread-pool hand-off, A2A-style JSON-RPC on localhost HTTP, MCP tools/call as newline-delimited JSON-RPC on a socket pair with context in params._meta, a queue, and a webhook back to the orchestrator. That gives 48 spans and 21 context hops per request. The in-process failure is real: Python’s ThreadPoolExecutor.submit does not copy context variables unless the caller does. We removed context at one hop type at a time (20 requests each), switched the queue consumer to the messaging default of a link instead of a parent, and dropped each hop independently with probability 1 − p for p from 0.80 to 1.00 (200 requests each). A correlation ID in each payload lets us count, per request, spans in the root trace, spans reachable through parents or links, and distinct trace IDs.

Overhead. We measured the bytes that traceparent, then tracestate (the W3C example value) and then a three-entry baggage (agent ID, conversation ID, tenant) add as HTTP header lines, as MCP _meta JSON and as JSON queue headers. We timed SDK inject plus extract (median of 5 × 20,000 calls) and ran 2,800 interleaved localhost round trips per arm with and without full context, with the server extracting context on every request.

Sampling. A Monte Carlo of 20,000 requests through the same 48-span shape uses an assumed latency model (our scenario, not measured): agent reasoning with a lognormal median of 1.5 s, MCP calls 0.3 s, queue waits with a median of 1 s or 10 s, an optional 5% of jobs held 5 to 60 minutes for human approval, a 4 s worker LLM step and an exporter delay drawn uniformly from 0 to 5 s, matching the SDK’s default 5,000 ms batch delay. In 3% of requests one asynchronous LLM span fails. We compare parent-based head sampling, head sampling re-decided at each asynchronous hop, and a model of the collector tail sampler: a decision after decision_wait from the first span’s arrival, keep if any span seen has an error, otherwise keep with the base probability. The collector’s decision cache is inactive by default (size 0); we model late spans as following the cached decision when a cache is configured and otherwise as a new trace evaluated by the same policy, which is our modelling assumption[26]^[26].

Coverage matrix. We scored 13 documents only from the cited page, read on 21 October 2026, on three criteria: a cross-process carrier (0 none, 1 generic or vendor-specific, 2 W3C traceparent), agent and tool span semantics, and guidance for asynchronous or indirect hops. The file data/coverage.csv holds the URL and the evidence for every cell.

The setup follows research tools that pair OpenTelemetry tracing with fault injection in multi-agent systems (ICST 2026)[27]^[27] or use OpenTelemetry traces with mocked LLMs for structural tests of agents (IEEE BigData 2025)[28]^[28]. We measure the trace graph itself.

4. Results #

4.1 RQ1: one missing hop splits the request #

A missing hop near the root costs the most (chart 1, left). When the planner ran on a thread pool without a context copy, the root trace held only its own span, 2.08% of 48, and everything else formed one second trace. Missing A2A headers kept 8.33%, the orchestrator side, and produced 3 traces per request. A missing MCP _meta kept 25.0% and split the request into 7 traces, one per tool call. Missing queue headers kept 50.0%, and missing webhook headers 87.5%, but both still produced 7 traces. With no hop propagating, one request became 22 traces.

Configuration (20 requests each)	Spans in root trace, %	Reachable via parent or link, %	Traces per request
All hops propagate	100.0	100.0	1
Thread pool without context copy	2.08	2.08	2
A2A HTTP headers missing	8.33	8.33	3
MCP params._meta missing	25.0	25.0	7
Queue message headers missing	50.0	50.0	7
Webhook headers missing	87.5	87.5	7
Queue consumer links instead of parenting	50.0	100.0	7
No hop propagates	2.08	2.08	22

Source: our experiment, results.json (OpenTelemetry Python SDK 1.45.1, 48 spans per request).

The link row matters for practice. Following the messaging convention’s default[11]^[11] left only half the spans in the root trace, but every span stayed reachable once links were followed. Only a backend that traverses links will show it as one request. Annotrace’s need to read six trace formats, including OpenTelemetry GenAI, LangSmith and Langfuse, shows how far tools are from a shared view of one run[29]^[29].

Random gaps compound (chart 1, right). If each of the 21 hops carries context with probability p, the chance that a request stays whole is p^21, and the measurements tracked it: 11.0% of requests were fully connected at p = 0.90 (analytic 10.94%), 39.0% at 0.95 (34.06%), 65.5% at 0.98 (65.43%) and 83.5% at 0.99 (80.97%). The share of spans in the root trace degrades more gently, from 97.11% at 0.99 to 73.8% at 0.90, since most hops sit near the leaves. Span counts therefore hide the problem: 97% of spans attached can coexist with one broken request in six. At the “thousands of inter-agent interactions per hour” that GAAT attributes to enterprise systems[30]^[30], that is many broken requests.

4.2 RQ2: what carrying context costs #

The bytes are small in absolute terms and large relative to tool calls (chart 2). A traceparent alone added 70 bytes as an HTTP header line, 82 bytes in MCP _meta and 71 bytes as a queue header. With tracestate the figures were 123, 137 and 126 bytes, and with baggage 232, 248 and 237. Measured against SEP-414’s example tools/call, which is 114 bytes in compact JSON, full context more than tripled the message, an increase of 217.5%. Against a minimal 316-byte A2A request it added 73.4%.

Carrier	Baseline message, bytes	+ traceparent	+ tracestate	+ baggage	Added, % of baseline (full context)
HTTP headers (A2A request, webhook)	316	70	123	232	73.4
MCP params._meta (JSON-RPC)	114	82	137	248	217.5
Queue message headers (JSON)	66	71	126	237	not computed (synthetic body)

Source: our measurement, results.json; values from the W3C and OpenTelemetry examples[6]^[6][8]^[8].

CPU cost was in microseconds. Inject plus extract of traceparent took a median of 6.06 µs, and 20.72 µs with baggage, which is about as much as creating and ending one span (21.15 µs). End to end, full context raised the median localhost round trip from 470.5 to 540.0 µs over HTTP/1.1, an increase of 69.5 µs, and from 171.7 to 226.5 µs over the JSON-RPC socket pair, 54.8 µs. These deltas include server-side parsing and extraction in pure Python, and we expect compiled stacks to pay less; against LLM calls measured in seconds they are below 0.01% (our arithmetic: 69.5 µs against a 1 s call). The case for trimming context is bytes and exposure, not latency. ATP shows the scale of encoding savings available: it cut wire payload by 96.4% against OpenTelemetry JSON for telemetry export[31]^[31].

4.3 RQ3: sampling against asynchronous hops #

Head sampling decides at the root, before any failure exists, so it keeps failing traces at the base rate: 6.34%, 4.12% and 6.43% across the three scenarios at a 5% rate (about 600 failing requests, so within noise of 5%). If the sampled flag does not cross the queue and each asynchronous job decides again, no failing trace survived whole (0.0%) because seven independent decisions must all say keep.

Tail sampling depends on whether the trace has fully arrived when the timer fires (chart 3). With 1-second queue waits, spans of one request arrived over a 95th-percentile window of 28.7 s, and the default 30-second decision wait kept 96.75% of failing traces complete without a decision cache and 99.66% with one. With 10-second median queue waits the window grew to 117.3 s and the default wait kept only 22.30%, or 80.45% with a cache. Waiting 120 s restored 96.40% without a cache and 99.83% with one. When 5% of jobs waited for human approval, the window’s 95th percentile reached 3,059.2 s and even a 300-second wait kept 76.28% without a cache and 94.40% with one.

Strategy at a 5% base rate (queue median 10 s)	Complete failing traces, %	Error span stored, %	All spans stored, %
Head, parent-based	4.12	4.12	5.14
Head, re-decided per async hop	0.0	3.60	4.89
Tail, wait 30 s, no decision cache (default)	22.30	100.0	7.21
Tail, wait 30 s, decision cache	80.45	80.45	7.28
Tail, wait 120 s, no decision cache	96.40	100.0	7.73
Tail, wait 120 s, decision cache	99.83	99.83	7.85

Source: our Monte Carlo, results.json; 20,000 requests, 3% failing; collector defaults from[26]^[26].

The cache trades one failure for another. Without it, a late span that carries the error forms a new trace that the error policy keeps, so every error span was stored, but its earlier context was usually dropped. With it, late spans follow the early decision, which keeps traces whole but loses the error when the early part looked healthy and was dropped: at a 30-second wait the error span was stored for 80.45% of failing traces, so 19.55% lost it. Longer waits cost memory: at the defaults of 50,000 traces in memory and a 30-second wait, one collector instance admits about 1,666.7 new traces per second before eviction (our arithmetic), and longer waits lower that ceiling.

5. Discussion #

The documents leave the riskiest hops unspecified. The coverage matrix (chart 4) lines up with the experiment. Carriers are specified for HTTP[6]^[6][10]^[10] and for MCP messages[7]^[7][8]^[8]. Only the messaging conventions treat asynchronous hops in full[11]^[11]. The A2A specification scores zero on all three criteria[9]^[9]. No document we scored specifies a carrier for A2A push-notification callbacks, and only one framework page mentions links across human-in-the-loop pauses. The top scores, 5 of 6, went to the OpenTelemetry MCP conventions and Microsoft Agent Framework, whose page names hosted tools as a boundary it cannot cross[12]^[12].

graph TD
    A[Hop crosses a process] --> B{Synchronous request}
    B -->|yes, HTTP| C[W3C headers]
    B -->|yes, MCP| D[traceparent in params meta]
    B -->|no| E{Queue, callback or pause}
    E --> F[Inject creation context and link consumer]
    F --> G{Trace arrival window over decision wait}
    G -->|shorter| H[Tail sampling with default wait]
    G -->|longer| I[Raise wait, add decision cache, keep error fragments]
    D --> J{Hosted tool runtime}
    J -->|yes| K[Correlate by conversation and task IDs]

What practitioners should do. First, treat propagation coverage as a metric with a target: at 21 hops, 99% coverage still breaks about one request in six, so the useful target is every hop, checked in CI by asserting one trace ID per request in tests, as OpenTelemetry-based structural testing allows[28]^[28]. Second, fix the thread-pool hand-off and _meta first: they cost the most continuity in our runs and are one-line fixes. Third, size tail sampling from the measured span arrival window, not from the default: set decision_wait above the 95th percentile of the window, enable the decision cache, and add a policy that keeps late error fragments. Fourth, keep a correlation ID such as A2A’s contextId[9]^[9] or the Agents SDK’s group_id[13]^[13] on every span, because hosted tools and human pauses will break parenting anyway.

Context is also a data leak and a trust input. Baggage that names agents, conversations and tenants crosses every hop it rides on. Communication metadata alone let a classifier recover a task’s class at 6 times chance on real A2A traffic[32]^[32], and a threat model of MCP, A2A, Agora and ANP identifies twelve protocol-level risks[33]^[33]. A census of 179 remote MCP endpoints found hosting concentrated (a Herfindahl-Hirschman Index of 0.736) and 95% of commercial platform-hosted servers behind gateway OAuth[34]^[34], so context increasingly crosses into infrastructure that the caller does not run. Gateways should strip context from untrusted callers and limit baggage to identifiers safe to expose. For audit, propagated IDs are where tamper-evident records attach: A2A-ForensicTrace verified signed evidence from 240 executions offline at a median of 1.61 ms per scenario trace[35]^[35], and the Agent Status Protocol proposal infers agent liveness from missed heartbeats rather than trusting self-reports[36]^[36].

Limitations and threats to validity. The chain runs in one process, with real sockets but threads instead of services, so the continuity results reflect propagation logic, not network faults. The sampling latency model is assumed, and the absolute capture rates move with it; the ranking of strategies and the role of the arrival window do not. Our tail-sampler model simplifies the collector: it ignores decision_wait_after_root_received, memory eviction and the span-ingest strategy. Latency deltas come from pure-Python stacks on a shared virtual machine and vary by tens of percent between runs. The coverage matrix scores single pages; a framework may document propagation elsewhere, and a gap analysis of MCP, A2A and ACP against governance requirements rates its own findings by how fast these protocols evolve, a reminder that such scores date quickly[37]^[37].

What would change the conclusion. If A2A adds a trace field to push notifications, or MCP hosts begin forwarding _meta context, the two largest unspecified boundaries close. A collector that defers decisions until a root span declares completion would weaken the case for long fixed waits.

6. Conclusion #

RQ1 Finding: Continuity loss depends on where the gap sits, and random gaps compound across hops. Measured by the share of a request’s 48 spans left in the root trace = 25.0% with MCP _meta missing and 2.08% with a thread-pool hand-off missing, and by fully connected requests = 39.0% at 95% per-hop coverage across 21 hops. This matters for the series because the failure-attribution methods discussed here assume a whole trace, and span-level dashboards hide the breakage.

RQ2 Finding: Carrying full context is cheap in time and noticeable in bytes on small tool calls. Measured by added bytes per message = 232 to 248 and added localhost round-trip time = 54.8 to 69.5 µs, an increase of 217.5% on a minimal MCP tools/call. This matters for the series because cost is no argument against propagation; the reasons to trim baggage are size and exposure.

RQ3 Finding: Tail sampling keeps most complete failing multi-agent traces only when its decision wait covers the trace’s arrival window. Measured by complete failing traces kept at a 5% base rate with 10-second queue waits = 22.30% at the default 30-second wait, 80.45% with a decision cache and 99.83% at 120 s with a cache, against 4.12% for parent-based head sampling. This matters for the series because the retention and cost choices compared in the previous article only pay off if the stored traces are complete.

The next question in the series is how to measure semantic failures, a wrong answer with a success status, which neither propagation nor error-based sampling can see.