In one sentence. A single request passes through five to seven internal components, and tracing attributes every microsecond to one of them — so "it's slow" becomes "it's slow here."
Picture it like this. A package tracking number. Without one, a late delivery is a mystery you argue about with customer service. With one, you can see the parcel sat in a depot for eleven hours, and the argument turns into a fix.
Why it matters. This is the least glamorous infrastructure in the series and the reason everything else in it can be debugged at 3am rather than guessed at. It also carries a privacy wrinkle unique to a TEE system: trace data, by design, leaves the enclave.
A single ComputeFlux gateway request touches five to seven internal components — HTTP middleware chain, rate limiter, API key resolver, scheduler, protocol adaptor, HTTP client to provider — and each component can be a bottleneck. Without distributed tracing, latency analysis degrades into guesswork: is the scheduler slow because the provider filter is scanning too many PebbleDB keys, or because the circuit breaker is holding connections open against a degraded provider? Tracing answers this by attributing every microsecond of wall-clock time to a specific span in a specific component, producing a causal graph of the request's execution timeline. ComputeFlux's tracing architecture builds on OpenTelemetry's span model, extended with AI-gateway-specific attributes at the scheduler layer — the request-routing and provider-call path where most of that latency variance actually lives, distinct from the on-chain consensus path covered in earlier articles.
The last two articles were about oversight and trust. Article 18 and Article 19 both asked whether we can verify the system is behaving correctly.
This one asks something related but distinct: can we see what the system is doing, moment to moment, well enough to operate it? That shifts from the trust thread to the operational thread — cross-cutting infrastructure that every other component in the series depends on for debuggability, from the scheduler to the relay to the settlement pipeline. It's a natural pivot. The next few articles are about running ComputeFlux at scale rather than proving it trustworthy.
Span Context Propagation and the Scheduler's Observability Layer
A span is a named, timed operation with a start time, an end time, and a set of key-value attributes. In ComputeFlux, the scheduler package (pkg/services/scheduler) owns an Observability struct wrapping an OTel trace.Tracer and metric.Meter, both pulled from the global otel.GetTracerProvider() and otel.GetMeterProvider().
That indirection matters. Instrumentation code only ever asks the global API for a tracer or meter; it never constructs an exporter or backend client itself. So whoever wires up the process decides where trace and metric data goes — including an operator who leaves it unconfigured and gets OTel's no-op provider. None of it touches the call sites.
Within the gateway process, context propagation is handled by Go's context.Context, and a call through the scheduler creates a span for the outbound call to a provider endpoint:
ctx, span := o.tracer.Start(ctx, SpanEndpointCall, trace.WithSpanKind(trace.SpanKindClient))
span.SetAttributes(
attribute.String(AttrGenAIProvider, string(ep.Provider)),
attribute.String(AttrGenAIRequestModel, string(model)),
attribute.String(AttrSchedEndpointKey, ep.KeyID),
attribute.Int(AttrSchedAttempt, attempt),
)
SpanEndpointCall (sched.endpoint.call) is the scheduler's primary span today. It wraps a single attempt at calling a provider endpoint, tagged with the provider, the requested model, the truncated key ID, and the retry attempt number. Its kind is explicitly Client, the correct OTel convention for an outbound call to another service.
A fuller span tree would go further: child spans for request validation, API key resolution, and individual storage reads, plus W3C Trace Context (traceparent) propagation across the HTTP boundary to upstream providers. That's a natural next step, and the current instrumentation doesn't implement it. Today the unit of observability is the endpoint call itself.
The attribute naming mixes two namespaces deliberately: standard-ish gen_ai.* keys (gen_ai.provider.name, gen_ai.request.model) that align with the emerging OTel semantic conventions for generative AI workloads, and a custom sched.* namespace (sched.endpoint.key_id, sched.attempt) for scheduler-specific concepts that don't have an existing convention to borrow. This produces a natural grouping in trace visualization tools (Jaeger, Grafana Tempo, Honeycomb): every sched.*-tagged span or metric clusters together, while the gen_ai.* attributes stay interoperable with any other tool that understands the same convention.
Sampling Strategy: Head-Based vs Tail-Based
Tracing every request would produce an unmanageable volume of data at real gateway scale — enough spans per second that storing, transmitting, and querying all of it would overwhelm any tracing backend not designed for that volume. Sampling reduces the volume by recording only a fraction of traces, and there are two fundamentally different ways to decide which fraction.
Head-based sampling makes the sampling decision at the start of the trace, before any spans are recorded. The root span's creation includes a sampling decision (typically a fixed probability), and this decision propagates to all child spans. If the root span is not sampled, no child spans are recorded. Head-based sampling is simple, stateless, and requires no coordination between services — each service independently respects the sampling flag in the trace context.
The weakness of head-based sampling is that it's indiscriminate. A low sample rate means most of the interesting traces — the ones with high latency, errors, or unusual patterns — are discarded before they can be analyzed, purely by bad luck, alongside all the boring ones.
Tail-based sampling defers the sampling decision until the trace is complete. All spans are recorded in memory (or a local buffer), and after the root span ends, a decision is made based on the trace's characteristics: keep all traces with high latency, all traces with errors, and a random sample of remaining traces. Tail-based sampling ensures that a much higher share of interesting traces are retained while still reducing total volume — at the cost of having to buffer every in-flight trace until it completes, which is a real memory cost for long-running streaming requests.
ComputeFlux hard-codes neither strategy. The Observability layer emits spans and metrics through the standard OTel API and leaves sampling to whatever TracerProvider the process is configured with — head-based sampling being the SDK's default when nothing else is specified.
Because the instrumentation talks to the API rather than a backend, adopting a tail-based or hybrid sampler later means configuring the TracerProvider, not rewriting spans. A self-hosted deployment or a managed backend like Grafana Cloud can make that call independently.
Worth being clear about one thing: the hybrid policy sketched above — force-sampling slow or erroring traces, a fixed base rate, a bounded in-memory buffer — is what a production deployment would configure at the collector or backend layer. It isn't baked into ComputeFlux's code today.
Why OpenTelemetry Over Jaeger/Zipkin Native SDKs
The choice of OpenTelemetry is not about features — Jaeger and Zipkin both support distributed tracing, span context propagation, and sampling. The choice is about vendor neutrality and future-proofing.
OpenTelemetry is the CNCF standard for observability data. It defines a unified API and SDK for traces, metrics, and logs, with a plugin architecture for exporters. An application instrumented with OpenTelemetry can export to Jaeger, Zipkin, Prometheus, Grafana Cloud, Datadog, Honeycomb, or any OTLP-compatible backend by changing one configuration line — the exporter. An application instrumented with Jaeger's native SDK is locked into Jaeger's backend (or must endure a migration that touches every instrumentation call site).
For ComputeFlux, the practical payoff is deployment flexibility. A self-hosted community deployment might pick Jaeger for its simplicity and cost — open source, single binary. A managed cloud deployment might pick Grafana Cloud for integrated dashboards and alerting. That distinction matters across the range of topologies Article 21 covers, from a developer's laptop to a Kubernetes Deployment on hardware SGX. The instrumentation code is identical in every one of them. This is particularly valuable for a TEE-based system where rebuilding and re-attesting the enclave binary is a governance-gated process — changing the observability backend should not require changing the attested code.
The additional benefit is the unified semantic convention. OpenTelemetry defines standard attribute names for HTTP requests (http.method, http.status_code, http.url), RPC calls (rpc.service, rpc.method), and databases (db.system, db.operation). By adhering to these conventions, ComputeFlux's spans are directly interpretable by any OTel-compatible analysis tool without custom mapping. Jaeger native spans might use http.status_code or http.status or statusCode depending on the instrumentation library — OpenTelemetry's semantic conventions eliminate this ambiguity.
The cost is a slightly heavier SDK — the OTel Go SDK's trace and metric providers add binary size and a small amount of per-span overhead (sampling decision, context propagation, attribute handling). This is negligible for an AI gateway where the dominant latency contributors are network round-trips and LLM inference itself, both of which run from milliseconds to tens of seconds — orders of magnitude larger than any per-span instrumentation cost.
Metric Types: Counter for Volume, Histogram for Distribution
Tracing answers "what happened during this specific request?" Metrics answer "what is happening across all requests?" They serve different analytical purposes, and the metric types must match the questions.
Counters are monotonically increasing values tracking cumulative totals: requests dispatched, successes, errors, retries, fallbacks. Counters are the right type here because they're immune to sampling artifacts. Metrics are separate from trace sampling, so the counter increments on every request and reflects the true total even when only a fraction of traces are recorded.
ComputeFlux's scheduler exposes a family of them — sched.requests.total, sched.requests.success, sched.requests.error, sched.retries.total, sched.fallbacks.total — each carrying provider, model, and error class attributes where relevant. Query sum(sched.requests.error{sched.error.class="rate_limit"}) and you get every rate-limited request across all providers, sampling notwithstanding.
Histograms capture the distribution of a value across many observations: request latency, time-to-first-token, and so on. A histogram records each observation into a set of buckets, incrementing a counter for the matching bucket. From bucket counts, any percentile can be estimated: P50, P95, P99. ComputeFlux's scheduler exposes two: sched.request.duration, end-to-end request duration in milliseconds, and sched.stream.ttft, time to first streaming chunk. Neither pins its bucket boundaries in the instrumentation code. They come from whichever OTel metric reader or exporter the deployment configures. So the same instrumentation can be re-bucketed for a completely different latency profile — fast embedding calls versus long-running chat completions — without redeploying the gateway binary.
Histograms are the metric type most sensitive to bucket configuration in general: whatever the highest configured bucket boundary is, any observation above it collapses into the same "+Inf" bucket, and no percentile above that boundary can be recovered from the data. For a system with request latencies ranging from sub-second calls to long streaming responses, the bucket boundaries need to be chosen — or the exporter configured — with that full range in mind.
Attributes: What Gets Tagged and Why
Span attributes are the dimensions by which traces are filtered and aggregated. Adding an attribute to every span increases the amount of data per span but enables queries that would otherwise require re-instrumentation. The attribute selection is a balance: too few attributes and traces are unqueriable; too many and trace storage costs explode.
ComputeFlux's scheduler tags its endpoint-call span, and the metrics recorded alongside it, with a mix of OTel GenAI semantic-convention attributes and its own sched.* namespace:
gen_ai.provider.name(string): the provider name (e.g.,openai,anthropic). This is the standard OTel attribute for "which AI provider handled this," so it stays interoperable with any OTel-aware tool.gen_ai.request.model(string): the requested model name. Combined with the provider attribute, this enables latency comparison across models and providers.sched.endpoint.key_id(string): the endpoint's key identifier. This enables per-key usage tracking without exposing the full API key in trace or metric data.sched.attempt(int): the retry attempt number (0 for first try, incrementing on each retry/fallback). A high average attempt count for a provider indicates elevated error rates.sched.is_fallback(bool): whether this attempt is a cross-model fallback rather than a same-model retry.sched.error.class(string): the classified error type used when recording a failed request, enabling alerting rules like "page the on-call engineer if the error rate exceeds a threshold for any provider."
The key-ID attribute deserves a privacy note whatever the tagging scheme. Full API keys must never appear in trace or metric data, because that data can end up in third-party SaaS platforms — Honeycomb, Datadog — well outside the TEE trust boundary. Any key-derived attribute has to be reduced to something sufficient for correlation, so requests using the same key share a tag value, and insufficient for reconstructing the key.
Key Takeaways
- Without tracing, latency analysis is guesswork dressed up as engineering. With it, you get a causal timeline that assigns time to specific components — and arguments become measurements.
- The instrumentation never names a destination. Code asks a global provider for a tracer; whoever runs the process decides where the data lands, including nowhere at all. Call sites never change when the backend does.
- Sampling is a real trade-off, not a tuning detail. Recording everything is expensive; recording a fraction risks missing rare events — and the interesting failures are precisely the rare ones.
- A privacy constraint specific to this system: trace data typically ends up in third-party services outside the enclave's trust boundary. Anything key-derived must be enough to correlate requests and never enough to reconstruct a key.
- Standard attribute names matter more than they look. They're what makes this data legible to tools that haven't been written yet.
From Visibility to Deployment
Tracing is what makes every other subsystem in this series — the scheduler, the relay, the billing pipeline, the contract runtime — debuggable in production rather than opaque. But visibility only matters if there's a production to observe.
Next — Article 21: Deployment Architecture: how all of this is actually packaged and run, from three nodes on a laptop to a fleet of hardware enclaves.