Tracing a Request Across Queue Boundaries with OpenTelemetry

Synchronous tracing is a solved problem — install the OTel SDK, auto-instrument HTTP and your database driver, and waterfalls appear. Then a request dispatches a queue job, and the trace just stops. The user-facing symptom ('my export never arrived') lives three async hops away from the request that triggered it, and your tracing backend shows you a 40ms POST that did nothing wrong. In queue-heavy systems — which is to say, in every serious backend I've worked on — the async gap is where debugging time actually goes.

Context doesn't cross the queue unless you carry it

OTel context propagation works over HTTP because instrumentation injects traceparent headers automatically. Message queues have no universal equivalent, so you do it explicitly: serialize the current context into message attributes at dispatch, extract it in the worker, and start the consumer span inside that restored context. The W3C Trace Context format is the same one HTTP uses — the queue message is just another carrier.

from opentelemetry import trace, propagate

tracer = trace.get_tracer("worker")

def dispatch(queue, task_name, body):
    headers = {}
    propagate.inject(headers)  # writes traceparent/tracestate
    queue.publish(task_name, body, attributes=headers)

def handle(message):
    parent_ctx = propagate.extract(message.attributes)
    with tracer.start_as_current_span(
        f"process {message.task_name}",
        context=parent_ctx,
        kind=trace.SpanKind.CONSUMER,
        attributes={
            "messaging.system": "rabbitmq",
            "messaging.destination.name": message.queue,
            "messaging.message.id": message.id,
        },
    ):
        run_task(message)

Twenty lines, and the trace now reads: HTTP request, PRODUCER span for the publish, queue wait, CONSUMER span for the processing, and every database call the worker made — one waterfall, end to end.

Child spans vs. span links: choose deliberately

Making the consumer span a child of the producer works beautifully for one-job-per-request flows. It breaks down for batch consumers and fan-in: a worker that aggregates 500 messages can't be a child of 500 traces. That's what span links are for — start a new trace for the batch and link each source context. My rule of thumb: parent-child when one message means one unit of user-attributable work; links when messages are aggregated, and for very long-delayed jobs where a 6-hour trace would be unreadable anyway.

Queue wait time is the span you're missing

The gap between the producer span ending and the consumer span starting is time spent sitting in the queue — and it's the metric that explains most 'the system is slow' reports in async pipelines. Record publish time as a message attribute and emit the delta as both a span attribute and a histogram metric. When exports get slow, this number tells you instantly whether to look at the worker code or the worker count.

• Propagate context through every hop, including job-dispatches-job chains — one missing link severs the trace.
• Use messaging semantic conventions so backends recognize and visualize queue hops correctly.
• Record retry attempt number on consumer spans; a trace showing attempt 3 of 3 tells a different story than attempt 1.
• Use parent-based tail-aware sampling where you can — head sampling at the HTTP edge will randomly orphan worker spans.

An observability setup that ends at the queue boundary observes the part of the system that was already easy to debug.

Payoff: incidents become queries

After we wired propagation through every queue in one platform, the flagship debugging moment came within a month: a customer's invoices were arriving hours late. One trace query showed a 3-hour gap between producer and consumer spans on a single queue — a supervisor misconfiguration had starved one worker pool while dashboards averaged the healthy ones. Pre-tracing, that was a day of log archaeology across four services. With the trace, it was fifteen minutes, most of which was writing the fix.