Distributed Tracing Lifecycle Visualization
Walk through the Distributed Tracing lifecycle with guided steps for trace id, span id, context propagation, waterfall, sampling.
Continue with the guided workspace
Use the paired workspace to connect this guide's mental model with live component state, visible signals, and the outcome each step or scenario is meant to prove.
Distributed tracing follows a single request as it fans out across many microservices, message queues, and databases, stitching the pieces back into one end-to-end timeline. Each hop becomes a span, all spans share a trace ID, and the resulting waterfall shows where time is actually spent — which is hard to see from any single service’s logs.
For parameter experimentation and stress-testing, the Distributed Tracing Simulator is also available.
Mental Model
Distributed tracing tracks the lifecycle of a request as it flows through multiple physical services, message queues, and databases.
- Trace: Represents the entire end-to-end journey of a request.
- Trace ID: A globally unique identifier shared by every microservice participating in a single request transaction.
- Span: Represents a single unit of work within a service (e.g., an HTTP handler call or a database query).
- Span ID: A unique identifier for a specific span.
- Parent ID: Identifies the span that triggered the current span, allowing tracing UI engines to construct a tree-like hierarchy (the waterfall view).
When a service makes an external call, it injects the current trace context into request headers so that downstream services can read the parent span ID and trace ID, stitching the end-to-end flow together.
Components and Why They Exist
Client
The client application initiates user operations and dispatches the opening HTTP request toward the edge API Gateway. Each trace walkthrough starts when a request particle leaves the Client card for the gateway.
API Gateway
The API gateway serves as the edge entry point that mints the globally unique Trace ID, opens the root span, and forwards the request downstream. Its card displays the newly generated root span alongside the injected trace context headers.
Trace Context
The trace context carrier serializes the trace_id, parent_id, and sampling flags into W3C traceparent headers so downstream services can link their local work to the active trace. You can follow the traceparent badge as it rides along every outbound network connector.
Catalog Service
The catalog service processes product listings and fans requests out to downstream payment and inventory dependencies. Upon extracting the incoming header context, its card opens child span span_id: sp-catalog linked to the gateway’s root span.
Payment Service
The payment service isolates card authorization and billing checks within its own child span. When invoked by the catalog service, it opens a payment span, queries the database, and records its execution latency before returning.
Inventory Service
The inventory service verifies item stock levels concurrently alongside payment processing. Its span bar runs in parallel with the payment service span under the shared catalog parent, illustrating concurrent fan-out work.
Database
The database persists service state and executes SQL queries whose durations are captured as leaf client spans. Slow queries on the Database card stretch the corresponding SQL execution span and visibly lengthen the parent service timeline.
Collector
The trace collector receives raw span batches emitted asynchronously by all microservices and buffers them before indexing. Span particles stream into the Collector card’s memory buffer queue as each service finishes its work.
Sampling Gate
The sampling gate filters high-volume routine traces so observability storage costs stay bounded while preserving slow or errored requests. Depending on whether head-based or tail-based rules are active, the gate badge marks each trace as sampled or dropped.
Trace Store
The trace store durably persists kept span records and indexes them by trace_id, service name, duration, and error tags. Sampled span batches flush from the gate into the Trace Store card to assemble the complete trace record.
Waterfall View
The waterfall view reconstructs the parent-child span hierarchy on a shared horizontal timeline so engineers can spot critical-path bottlenecks at a glance. Nested Gantt bars on the waterfall panel highlight which service or database hop dominated end-to-end latency.
Alerting Node
The alerting node evaluates trace latency and error-rate telemetry against service level thresholds to page operators when dependencies degrade. When a slow downstream span pushes total duration past the SLA limit, the alerting card flips to a warning state.
Edge Cases and Misconceptions
- Broken context propagation: If any service in a deep microservices chain is not configured to propagate context (e.g., a library update omits the outbound HTTP header middleware), the trace is broken. The downstream spans appear as separate root traces, creating fragmented graphs that make end-to-end debugging impossible.
- Clock skew in waterfalls: Span start/end timestamps are recorded locally by each service. If backend servers have drifted clocks (even by 10-20 milliseconds), waterfall UI builders can display negative latency spans, showing child spans starting before their parents did. Network time sync (NTP) is crucial.
- Asynchronous tasks as children: Developers often make async worker threads children of the triggering HTTP route. However, if the HTTP call returns early and the worker runs in the background, a standard child span will falsely inflate the parent HTTP span’s duration. Using Span Links instead of direct parent-child references correctly models async handoffs.
- Sampling bias: Relying purely on head-based sampling means a slow request or database error occurring late in the call path might be dropped if the gateway decided not to sample that trace ID at the request’s start. Tail-based sampling resolves this but introduces memory overhead on collectors.
Frequently Asked Questions
What makes spans part of one trace?
Every participating span carries the same trace ID. Parent span IDs record the causal relationship, while the trace ID keeps the complete request searchable as one timeline.
Why are payment and inventory shown in parallel?
The catalog service can start both child operations without waiting for either result. Their spans share a parent, but neither is the other’s parent, so the waterfall can show concurrent work rather than a false serial dependency.
Common Questions
How is context propagated across network boundaries?
Context propagation transfers tracing metadata (such as Trace ID, Span ID, and flags) across network boundaries using standardized protocols like the W3C Trace Context specification. The metadata is injected into outgoing HTTP request headers (e.g., as a `traceparent` header) or message metadata, and extracted by downstream services to establish parent-child relationships.
What is the difference between head-based and tail-based sampling?
Head-based sampling makes the sampling decision at the very beginning of a request's lifecycle (e.g., at the API Gateway), dropping or keeping the entire trace before knowing if it will succeed or encounter latency. Tail-based sampling buffers all spans for a trace at the collector layer and evaluates the complete trace before saving it, ensuring that rare errors and slow requests are always retained while normal traces are sampled down.
How do asynchronous message queues affect parent-child trace spans?
Asynchronous messaging breaks the synchronous request-response flow. Making a consumer process span a direct child of a producer span would incorrectly suggest the producer blocked waiting for the consumer. Instead, standard practices like OpenTelemetry use Span Links, referencing the producer's trace/span context to show correlation without implying a synchronous parent-child execution block.
Why can a trace look healthy while users still experience slowness?
A trace only explains the sampled request path that was captured. If sampling drops slow traces, client-side time is missing, or queue wait time is not instrumented, the waterfall can understate real user latency. Good tracing pairs spans with metrics, logs, and sampling rules that preserve errors and high-latency requests.