In one sentence. ComputeFlux sorts every failure into a category, and each category gets a different response — retry shortly, switch credentials, give up instantly, or stop calling that provider at all for a while.
Picture it like this. A receptionist redialing a number. A busy signal means try again in a moment. A disconnected number means stop and reach for a different contact. A wrong number means the mistake is yours, and redialing will never fix it. Systems that treat all three the same either give up too early or hammer a line that is never going to answer.
Why it matters. In a gateway sitting in front of many AI providers, failure isn't an exception — it's the steady state. Rate limits, expired credentials, and regional outages arrive constantly. The quality of a gateway is mostly the quality of its failure handling, which is why this is the longest article in the series.
Skip to the circuit breaker if you already know your retry taxonomy.
Article 8 covered how the Scheduler picks a provider endpoint and pins a session to it for cache-friendly routing. That's half of scheduling. The other half is what happens when the endpoint fails.
Upstream AI services are unreliable by nature. Rate limits, expired credentials, transient overloads, and outright outages aren't exceptions in a multi-provider gateway — they're the steady state. ComputeFlux answers with four layers: error classification, per-class retry strategies, circuit breaking, and on-chain failure accounting. Each addresses a different failure mode, and together they produce resilience no single mechanism could deliver alone.
This is also the retry logic that gets reused later, at a very different time scale, by cross-chain settlement — the subject of Article 11.
The Classify Decision Tree
The Classify function is a pure function from error to ErrorClass, with no side effects and no state. This purity matters: classification must be deterministic and reproducible, because the Scheduler's retry logic depends on it. The decision tree has four levels of precedence:
Level 1 — Context cancellation (highest priority): If the error wraps context.Canceled or context.DeadlineExceeded, it is immediately classified as ClassCancel. This check must come first because a canceled context can manifest as any underlying error type — a network timeout, a partially-read response body, a connection reset. Without this check, a client disconnecting mid-request would be misclassified as ClassTransient, triggering unnecessary retries that waste provider capacity.
Level 2 — HTTP status code from UpstreamError: The UpstreamError type carries an HTTP status code from the provider's response, classified along HTTP semantics:
429 Too Many Requests→ClassRateLimit: the provider is telling us to slow down. This is actionable — the Gateway can switch to a different key or wait and retry.401 Unauthorized,403 Forbidden,402 Payment Required→ClassAuth: the credential is invalid, expired, or the billing account is delinquent. Retrying with the same key is futile; the key must be blacklisted. The402inclusion is deliberate — in the context of paid API services it specifically signals an insufficient-funds billing account, a permanent credential problem rather than a transient one.5xx(500–599) →ClassTransient: the provider is experiencing an internal error that may resolve on its own. Retry with backoff is appropriate.4xx(400–499, excluding the above) →ClassClient: the request itself is malformed. Retrying will not help, because the error is in the request, not the provider.
Level 3 — Text-based heuristics: Some providers embed rate-limit information in the response body rather than in the HTTP status code. isRateLimitText checks for substrings like "rate limit", "rate_limit", "quota exceeded", "too many requests" — case-insensitively, via a manual byte-level toLower rather than importing strings, using substring matching rather than regex for predictable performance. This catches misbehaving proxies and non-standard implementations.
Structural connection failures fold into this same layer: DNS resolution failures, connection refused, TLS handshake failures. isNetworkError matches Go's standard-library error strings — "connection refused", "connection reset", "no such host", "timeout", "tls: handshake", "eof", "broken pipe", "i/o timeout", "dial tcp" — and classifies them as ClassTransient, the same class as a 5xx.
So an unreachable endpoint is not treated as permanently dead the way a bad credential is. It takes the same backoff-and-retry path as any other transient error, on the reasonable theory that a DNS blip or a one-off TLS handshake failure may well clear by the next attempt. Matching on message substrings is fragile, since the text can shift between Go versions, but it's practical: type-asserting against the public net.OpError type is more precise but couples the classifier to specific Go stdlib error shapes, which can also shift between versions.
Level 4 — Unknown (fallthrough): Anything unmatched becomes ClassUnknown, which the Scheduler treats as non-retryable by default. That's deliberately conservative. Retrying blind risks compounding a problem nobody understands yet — if the real cause is a malformed request, every attempt fails identically and the retries buy nothing but load.
Per-Class Retry Strategy
Each error class triggers a distinct action in the tryOneModel retry loop:
ClassRateLimit: the current key is added to usedKeys (not deadKeys — the key is still valid, just temporarily exhausted). The Scheduler backs off, then continues the loop; Select excludes usedKeys from consideration, so the next attempt uses a different key if one is available. If all keys for the endpoint are used, the endpoint is skipped and the next endpoint in the route is tried.
ClassAuth: the key is added to deadKeys. Unlike usedKeys, dead keys are never reconsidered during this request, even if the Scheduler exhausts all other options. The retry loop continues immediately without backoff — there's no point waiting to retry a credential that will still be dead in 200ms. (Structural connection failures stay in ClassTransient, so an unreachable endpoint still gets backoff-and-retry rather than being written off.) If all endpoints have all keys dead, the Scheduler returns ErrAllEndpointsDead, which propagates as a 503 to the client.
ClassTransient: the Scheduler backs off and retries the same endpoint. This is the most common retry path. The same key is retried — transient errors are typically server-side and affect all keys equally, so switching keys is unlikely to help.
ClassClient: no retry. The error returns immediately as a SchedError with the original status code, so the Gateway doesn't waste retries on a malformed request.
ClassCancel: no retry. The error returns as ErrCanceled, which the Gateway handler detects and does not count as a provider failure — the fault is the client disconnecting, not the provider failing.
The usedKeys versus deadKeys distinction is the crux of the design. Picture a provider with three API keys, all rate-limited at once because the account-level limit is blown. Key one returns 429 and goes to usedKeys. Key two, the same. Key three, the same — and now every key is used.
The Scheduler does not mark the endpoint dead. It simply has no keys left for this request. A request arriving a few seconds later tries all three again, because usedKeys is per-request state that never persists. That's the correct behavior: rate limits are time-windowed and reset on their own, while a dead key stays unusable for the life of the credential.
The Backoff Algorithm
The core mechanism is exponential growth with multiplicative jitter:
delay = min(initial * 2^attempt, maxDelay)
sleep = delay * random(jitterMin, jitterMax)
initial and maxDelay are configurable, and jitter is a random multiplier drawn from [jitterMin, jitterMax].
Two implementation details are worth noting. The backoff function never calls time.Sleep. It uses time.NewTimer with a select over both the timer channel and ctx.Done(), so a client canceling mid-wait interrupts the backoff instead of sleeping it out, and a deferred t.Stop() keeps the timer from leaking when that happens. And the exponential term uses a left-shift, initial << attempt, rather than math.Pow. A bit-shift is one CPU instruction; math.Pow pulls in floating-point work that's slower and, at the margin, not identical across architectures. At nanosecond scale on infrequent calls the difference hardly matters, but the shift is free.
ComputeFlux runs this same shape of backoff at two different tunings, because the two failure domains it serves have very different economics.
Scheduler-level (API calls): delay(n) = min(BackoffInitial × 2^n, BackoffMax) × U(0.8, 1.2), with BackoffInitial = 200ms, BackoffMax = 5s, MaxRetries = 3:
| Attempt | Base Delay | With Jitter Range | Cumulative (median) |
|---|---|---|---|
| 0 | 200ms | 160ms – 240ms | 200ms |
| 1 | 400ms | 320ms – 480ms | 600ms |
| 2 | 800ms | 640ms – 960ms | 1.4s |
Jitter here isn't decoration. It's what stops synchronized clients from retrying in lockstep. Without it, 100 concurrent requests that hit a rate limit at the same instant all retry exactly 200ms later, hit it again, retry 400ms later, and so on — permanently in step, never leaving the upstream limiter a gap wide enough to drain the backlog.
Add ±20% jitter and those 100 requests spread across an 80ms window. The limiter admits some from each wave and rejects others, and the queue gradually desynchronizes.
The ±20% figure is chosen, not arbitrary. Go wider, to ±50%, and you desynchronize faster but let some requests retry almost immediately, which defeats the point of backing off. Go narrower, to ±5%, and you barely desynchronize at all. The 80%–120% window is an equal-width multiplicative jitter, deliberately not AWS's "full jitter" (which draws each delay uniformly from zero to the cap, letting some requests retry almost immediately). The bounded window is wide enough to break lockstep within two or three cycles at typical concurrency, and narrow enough to preserve the spacing you asked for.
Settlement and relay syncs (a different failure domain): the retries covered in Article 10 and Article 11 — committing DKG-signed settlement batches on the TEE chain and syncing mainchain events down — don't use an exponential sequence at all.
Here's the mechanism. SyncToHub submits the aggregated threshold signature via chain.SignAndSubmit. On failure it calls HubSyncEnd(chainId, ids, false), which persists AsyncBatchState.LastSync = now - 300 seconds for that chain. IsHubSyncRuning counts a chain as "sync in progress" whenever now - LastSync <= 360, so backdating by 300 leaves the chain marked busy for roughly 60 more seconds. Then the flag flips to false and targetSyncRelay may start a fresh SyncTxStart for the same still-pending hub calls.
The effect is a flat ~60-second cooldown between attempts rather than a growing delay. There's no retry-count cap in this path either: a hub call gets re-attempted about once per cooldown window until it succeeds.
That coarser pause fits a failure domain where failure costs more and recovery takes longer than a single block. Settlement batches need a quorum of DKG signers online and responsive, and relayed events only become eligible once the mainchain's finality gadget has settled — seconds to tens of seconds on Substrate-based chains. Retrying on the millisecond cadence the API scheduler uses would be pure waste against a consensus or RPC hiccup. Sixty seconds is long enough for a brief blip or congestion event to clear, and short enough that a persistently failing sync shows up in logs and metrics within a few cycles instead of sitting silently stuck.
Circuit Breaker: The Third Layer
While the retry loop handles per-request failures, the Circuit Breaker handles persistent failures across many requests. Its state machine has three states:
Closed (normal operation): requests flow through normally. A background goroutine (RunTicker) periodically evaluates each endpoint's error rate. If (errorCount429 + errorCount5xx) / totalRequests >= failureRate and totalRequests >= minSampleSize, the breaker transitions to Open. The Tick method uses atomic swaps to read and reset counters simultaneously, preventing races where new requests arrive between reading TotalRequests and ErrorCount429. The sample window is effectively the interval between ticks (default: 5 seconds).
Open (failing): all requests are immediately rejected without contacting the upstream provider, preventing the Gateway from wasting resources on a known-broken endpoint and reducing load on the struggling provider. The breaker stays open for OpenDuration (default: 10 seconds) before transitioning to Half-Open.
Half-Open (probing): a limited number of probes, three by default, go through to test recovery. All succeed and the breaker closes. Any fails and it returns to Open for another OpenDuration. The limit is enforced by tracking hoSuccess + hoFailure rather than by capping concurrent probes, so a burst arriving while half-open lets only the first few through and rejects the rest until the probes resolve. That's intentional — flooding a recovering provider is the fastest way to trigger the next failure cycle.
The Allow method also checks metrics.CircuitOpen in addition to the breaker's own state. This dual-write is a backstop: even if the state machine has a bug — say, failing to transition from Half-Open to Closed correctly — the metrics.CircuitOpen boolean gives external monitoring something to observe and override.
Interplay Between the Layers
- Error Classification determines what kind of failure occurred.
- Retry Strategy determines how to respond to this specific failure — retry, skip, abort.
- Circuit Breaker determines whether to even attempt contacting the provider in the first place.
A typical cascade runs like this. The breaker is closed and requests flow, then the provider starts returning 503s. Each one classifies as ClassTransient, triggers a backoff-retry, and records a failure in the endpoint's metrics. Once minSampleSize requests have accumulated at or above the threshold failure rate, Tick opens the breaker. Subsequent requests are rejected immediately — no more 503s, no more wasted retries. Ten seconds later the breaker half-opens and probes. If the provider has recovered, the probes succeed, the breaker closes, and normal operation resumes.
The system degrades gracefully when a layer fails to help. If the breaker never opens because the failure rate stays below threshold, per-request retry still provides individual resilience. If the retry loop exhausts all keys and endpoints, ErrNoHealthyEndpoint or ErrAllEndpointsDead returns a clean 503 rather than a cryptic internal error.
Why PrepareProposal — Not a Timer Goroutine — Drives Settlement Retries
Cross-chain settlement retries are a special case worth calling out, because the trigger is architecturally different from the API-call retries above. Settlement retries are driven by PrepareProposal, the ABCI method that constructs the next block's transaction set — not by a separate goroutine on a timer. This isn't an implementation convenience, it's a correctness requirement.
Consider the alternative. A background goroutine polls the chain for transaction status and resubmits on failure. That races with block production. Suppose the goroutine resubmits a settlement transaction while PrepareProposal is building a block containing a conflicting state transition — the user spends the very credits the settlement was supposed to deduct. The resulting block is invalid. Preventing it would take a mutex between the retry goroutine and the consensus reactor, which defeats the point of separating them at all.
By embedding the retry logic in PrepareProposal, ComputeFlux serializes retry decisions through the same consensus pipeline as every other state transition:
PrepareProposalchecks (viaprepareRelayTxs/targetSyncRelay) for pending settlement syncs (DKG-signed batch submissions).- If a sync is already in progress (
IsHubSyncRuningreturns true within its 360-second window),PrepareProposalskips user transactions entirely and includes only system transactions — the sync continuation. This prevents ordering conflicts. - If no sync is in progress but hub calls are pending,
PrepareProposalinitiates a new sync via aSysCall_SyncTxStarttransaction. - The sync transaction goes through consensus like any other, with
FinalizeTxexecuting the actual submission and recording the result. - On failure,
HubSyncEnddeliberately backdates the chain'sLastSynctimestamp so thatIsHubSyncRuningkeeps reporting "busy" for roughly another 60 seconds, then clears — so the nextPrepareProposalthat observesIsHubSyncRuning() == falsere-triggers the same still-pending hub call.
So the cadence comes from that ~60-second cooldown, not from block time — far coarser than the millisecond-scale API backoff above. The risk profiles differ that much. A failed API call costs one user a few seconds of waiting. A failed settlement means a provider goes unpaid for inference work already delivered. Faced with that, ComputeFlux would rather wait out a short chain hiccup than hammer the chain every block.
Cleanup: Signature Data After Retry Exhaustion
When a settlement sync finishes — whether it succeeds or is retried indefinitely until it does — the cryptographic artifacts it generated have to be cleaned up. The sync collects DKG partial signatures from validators via sendPartialSignByIds, and those must be deleted afterward so they can't contaminate future transactions. DeleteSigOfTx(txn, txIndex) removes every key-value pair under prefix PartialSigPrefix + txIndex + "_", inside the same database transaction as the block commit.
That atomicity matters. If deletion and commit weren't transactional together, a node crash between them would leave orphaned partial signatures in PebbleDB. On restart, the node might fold those into a new aggregation and produce an invalid threshold signature the chain rejects.
Thanks to PebbleDB's LSM-tree structure, the prefix-scan deletion is O(n) in matching keys and O(1) in total database size. With 3–5 validators it's trivially fast. At 100+ it could become a bottleneck, and the fix would be a separate column family or a more granular composite key that supports range deletion.
Cleanup does one more thing. After a successful sync, HubSyncEnd refreshes AsyncBatchState.LastSync and clears the chain's sync state. That's what lets IsHubSyncRuning return false and PrepareProposal resume normal block production.
The UX Trade-Off
Every retry mechanism trades finality against latency. The scheduler's default — three retries starting at 200ms — puts worst-case latency around 1.4 seconds before the user sees a failure. For a chat completion API, where responses take 2–10 seconds anyway, that's fine. For a sub-500ms real-time application, it isn't.
Both knobs are per-deployment. A latency-sensitive deployment might run MaxRetries=1, BackoffInitial=50ms, trading success rate for speed. A batch-processing deployment might run MaxRetries=5, BackoffInitial=500ms, trading response time for eventual success.
Cross-chain settlement sits in a different category entirely. Its outcome is binary — paid or not paid — and a failed settlement costs more than a failed inference call. That's why it gets an indefinite, cooldown-gated loop instead of a small bounded retry count.
On-Chain Failure Accounting
Failures aren't just operational noise. They carry economic weight. SettlementWithRoute.FailCount tracks failures per SettlementWithRoute (each combination of user, provider, model, and route) within each settlement window, stored in PebbleDB and committed on-chain periodically, which turns them into an auditable SLA record. context.Canceled errors are deliberately left out, so nobody can damage a provider's reputation by connecting and immediately hanging up. What that record enables economically — SLA enforcement, routing reputation, user-side analytics — is Article 10's subject.
Observability as a Compensating Control
The scheduler exposes an OpenTelemetry retryTotal counter tagged by provider, so operators can watch retry rates live. A sudden spike on one provider is a leading indicator of an upstream outage, often visible before that provider's own status page catches up. A gradual rise across all of them suggests broader network degradation.
This is the practical complement to the backoff math above. Those exponential formulas assume failures are independent and randomly distributed. Real failures are frequently correlated — regional outages, provider-wide degradations. Retry-rate monitoring is what lets an operator tell "the math is working as expected" apart from "something is wrong that the math doesn't model."
Key Takeaways
- Four categories, checked in a strict order — and the order is load-bearing. A canceled request can disguise itself as almost any other error. Get the precedence wrong and a user closing their browser tab looks identical to a provider outage, triggering retries that waste real capacity.
- "Temporarily exhausted" and "permanently dead" are tracked separately. A rate-limited credential is set aside for this request and tried again on the next one. A rejected credential is never touched again during this request, no matter how few options remain.
- The randomness in the retry delay is doing real work. Without it, a hundred clients that fail at the same instant retry at the same instant, forever — and the provider's rate limiter never gets a gap wide enough to drain the backlog. A ±20% spread breaks the lockstep within two or three cycles.
- The circuit breaker sits above retries and asks a different question: not how to respond to this failure, but whether to place the call at all. Recovery is tested with a handful of probes, never by reopening the floodgates.
- Money retries differently from requests. A failed inference costs a user a few seconds. A failed settlement means a provider went unpaid for work already performed — so settlement retries indefinitely on a roughly 60-second cooldown instead of quitting after three tries.
- Failures become an auditable on-chain record. With one deliberate exclusion: client disconnects don't count, so nobody can damage a provider's reputation by connecting and immediately hanging up.
Error classification, retry strategy, and circuit breaking together embody a core principle of distributed systems: failures are inevitable, but cascading failures are preventable. By categorizing errors, capping retries, and isolating broken endpoints, ComputeFlux converts unpredictable upstream unreliability into predictable, bounded behavior — clients see either successful responses or clean error codes, never indefinite hangs or retry storms.
That closes out the execution layer. Requests now reach the right machine and survive the wrong one. Part 3 asks the question that turns all of this into a business rather than a demo: who pays whom, how much, and how would anyone prove it?
Next — Article 10: Billing & On-Chain Settlement: the failure counts this article records are one input to a bill. Here's the rest of it — and why a receipt you can verify beats an invoice you have to believe.