In one sentence. ComputeFlux sorts every failure into a category, and each category gets a different response — retry shortly, switch credentials, give up instantly, or stop calling that provider at all for a while.
Picture it like this. A receptionist redialing a number. A busy signal means try again in a moment. A disconnected number means stop and reach for a different contact. A wrong number means the mistake is yours, and redialing will never fix it. Systems that treat all three the same either give up too early or hammer a line that is never going to answer.
Why it matters. In a gateway sitting in front of many AI providers, failure isn't an exception — it's the steady state. Rate limits, expired credentials, and regional outages arrive constantly. The quality of a gateway is mostly the quality of its failure handling, which is why this is the longest article in the series.
Skip to the circuit breaker if you already know your retry taxonomy.
Article 8 covered how the Scheduler picks a provider endpoint and pins a session to it for cache-friendly routing. That's half of scheduling. The other half is what happens when the endpoint fails.
Upstream AI services are unreliable by nature. Rate limits, expired credentials, transient overloads, and outright outages aren't exceptions in a multi-provider gateway — they're the steady state. ComputeFlux answers with four layers: error classification, per-class retry strategies, circuit breaking, and on-chain failure accounting. Each addresses a different failure mode, and together they produce resilience no single mechanism could deliver alone.
This is also the retry logic that gets reused later, at a very different time scale, by cross-chain settlement — the subject of Article 11.
The Classify Decision Tree
The Classify function is a pure function from error to ErrorClass, with no side effects and no state. This purity matters: classification must be deterministic and reproducible, because the Scheduler's retry logic depends on it. The decision tree has four levels of precedence:
Level 1 — Context cancellation (highest priority): If the error wraps context.Canceled or context.DeadlineExceeded, it is immediately classified as ClassCancel. This check must come first because a canceled context can manifest as any underlying error type — a network timeout, a partially-read response body, a connection reset. Without this check, a client disconnecting mid-request would be misclassified as ClassTransient, triggering unnecessary retries that waste provider capacity.
Level 2 — HTTP status code from UpstreamError: The UpstreamError type carries an HTTP status code from the provider's response, classified along HTTP semantics:
429 Too Many Requests→ClassRateLimit: the provider is telling us to slow down. This is actionable — the Gateway can switch to a different key or wait and retry.401 Unauthorized,403 Forbidden,402 Payment Required→ClassAuth: the credential is invalid, expired, or the billing account is delinquent. Retrying with the same key is futile; the key must be blacklisted. The402inclusion is deliberate — in the context of paid API services it specifically signals an insufficient-funds billing account, a permanent credential problem rather than a transient one.5xx(500–599) →ClassTransient: the provider is experiencing an internal error that may resolve on its own. Retry with backoff is appropriate.4xx(400–499, excluding the above) →ClassClient: the request itself is malformed. Retrying will not help, because the error is in the request, not the provider.
Level 3 — Text-based heuristics: Some providers embed rate-limit information in the response body rather than in the HTTP status code. isRateLimitText checks for substrings like "rate limit", "rate_limit", "quota exceeded", "too many requests" — case-insensitively, via a manual byte-level toLower rather than importing strings, using substring matching rather than regex for predictable performance. This catches misbehaving proxies and non-standard implementations.
Structural connection failures fold into this same layer: DNS resolution failures, connection refused, TLS handshake failures. isNetworkError matches Go's standard-library error strings — "connection refused", "connection reset", "no such host", "timeout", "tls handshake", "eof", "broken pipe", "i/o timeout", "dial tcp" — and classifies them as ClassTransient, the same class as a 5xx.
So an unreachable endpoint is not treated as permanently dead the way a bad credential is. It takes the same backoff-and-retry path as any other transient error, on the reasonable theory that a DNS blip or a one-off TLS handshake failure may well clear by the next attempt. Matching on message substrings is fragile, since the text can shift between Go versions, but it's practical: type-asserting against internal net.OpError variants is more complex and just as version-dependent.
Level 4 — Unknown (fallthrough): Anything unmatched becomes ClassUnknown, which the Scheduler treats as non-retryable by default. That's deliberately conservative. Retrying blind risks compounding a problem nobody understands yet — if the real cause is a malformed request, every attempt fails identically and the retries buy nothing but load.
Per-Class Retry Strategy
Each error class triggers a distinct action in the tryOneModel retry loop:
ClassRateLimit: the current key is added to usedKeys (not deadKeys — the key is still valid, just temporarily exhausted). The Scheduler backs off, then continues the loop; Select excludes usedKeys from consideration, so the next attempt uses a different key if one is available. If all keys for the endpoint are used, the endpoint is skipped and the next endpoint in the route is tried.
ClassAuth (and dead structural endpoints): the key is added to deadKeys. Unlike usedKeys, dead keys are never reconsidered during this request, even if the Scheduler exhausts all other options. The retry loop continues immediately without backoff — there's no point waiting to retry a connection or credential that will still be dead in 200ms. If all endpoints have all keys dead, the Scheduler returns ErrAllEndpointsDead, which propagates as a 503 to the client.
ClassTransient: the Scheduler backs off and retries the same endpoint. This is the most common retry path. The same key is retried — transient errors are typically server-side and affect all keys equally, so switching keys is unlikely to help.
ClassClient: no retry. The error returns immediately as a SchedError with the original status code, so the Gateway doesn't waste retries on a malformed request.
ClassCancel: no retry. The error returns as ErrCanceled, which the Gateway handler detects and does not count as a provider failure — the fault is the client disconnecting, not the provider failing.
The usedKeys versus deadKeys distinction is the crux of the design. Picture a provider with three API keys, all rate-limited at once because the account-level limit is blown. Key one returns 429 and goes to usedKeys. Key two, the same. Key three, the same — and now every key is used.
The Scheduler does not mark the endpoint dead. It simply has no keys left for this request. A request arriving a few seconds later tries all three again, because usedKeys is per-request state that never persists. That's the correct behavior: rate limits are time-windowed and reset on their own, while a dead key stays unusable for the life of the credential.
The Backoff Algorithm
The core mechanism is exponential growth with multiplicative jitter:
delay = min(initial * 2^attempt, maxDelay)
sleep = delay * random(jitterMin, jitterMax)
initial and maxDelay are configurable, and jitter is a random multiplier drawn from [jitterMin, jitterMax].
Two implementation details are worth noting. The backoff function never calls time.Sleep. It uses time.NewTimer with a select over both the timer channel and ctx.Done(), so a client canceling mid-wait interrupts the backoff instead of sleeping it out, and a deferred t.Stop() keeps the timer from leaking when that happens. And the exponential term uses a left-shift, initial << attempt, rather than math.Pow. A bit-shift is one CPU instruction; math.Pow pulls in floating-point work that's slower and, at the margin, not identical across architectures. At nanosecond scale on infrequent calls the difference hardly matters, but the shift is free.
ComputeFlux runs this same shape of backoff at two different tunings, because the two failure domains it serves have very different economics.
Scheduler-level (API calls): delay(n) = min(BackoffInitial × 2^n, BackoffMax) × U(0.8, 1.2), with BackoffInitial = 200ms, BackoffMax = 5s, MaxRetries = 3:
| Attempt | Base Delay | With Jitter Range | Cumulative (median) |
|---|---|---|---|
| 0 | 200ms | 160ms – 240ms | 200ms |
| 1 | 400ms | 320ms – 480ms | 600ms |
| 2 | 800ms | 640ms – 960ms | 1.4s |
| 3 | 1.6s | 1.28s – 1.92s | 3.0s |
Jitter here isn't decoration. It's what stops synchronized clients from retrying in lockstep. Without it, 100 concurrent requests that hit a rate limit at the same instant all retry exactly 200ms later, hit it again, retry 400ms later, and so on — permanently in step, never leaving the upstream limiter a gap wide enough to drain the backlog.
Add ±20% jitter and those 100 requests spread across an 80ms window. The limiter admits some from each wave and rejects others, and the queue gradually desynchronizes.
The ±20% figure is chosen, not arbitrary. Go wider, to ±50%, and you desynchronize faster but let some requests retry almost immediately, which defeats the point of backing off. Go narrower, to ±5%, and you barely desynchronize at all. The 80%–120% window is the AWS-recommended "full jitter" approximation: wide enough to break lockstep within two or three cycles at typical concurrency, narrow enough to preserve the spacing you asked for.
Cross-chain settlement (a different failure domain): the retries covered in Article 11 — pushing settlement transactions from the sidechain up to the Polkadot relay chain — don't use an exponential sequence at all.
Here's the mechanism. SyncToHub submits the aggregated threshold signature via chain.SignAndSubmit. On failure it calls submitSyncTxEnd(chainId, ids, false), which persists AsyncBatchState.LastSync = now - 300 seconds for that chain. IsHubSyncRuning counts a chain as "sync in progress" whenever now - LastSync <= 360, so backdating by 300 leaves the chain marked busy for roughly 60 more seconds. Then the flag flips to false and targetSyncRelay may start a fresh SyncTxStart for the same still-pending hub calls.
The effect is a flat ~60-second cooldown between attempts rather than a growing delay. There's no retry-count cap in this path either: a hub call gets re-attempted about once per cooldown window until it succeeds.
That coarser pause fits a failure domain where failure costs more and recovery takes longer than a single block. Mainchain finality on Substrate-based chains runs seconds to tens of seconds, so retrying on the millisecond cadence the API scheduler uses would be pure waste against an RPC or consensus hiccup. Sixty seconds is long enough for a brief blip or congestion event to clear, and short enough that a persistently failing sync shows up in logs and metrics within a few cycles instead of sitting silently stuck.
Circuit Breaker: The Third Layer
While the retry loop handles per-request failures, the Circuit Breaker handles persistent failures across many requests. Its state machine has three states:
Closed (normal operation): requests flow through normally. A background goroutine (RunTicker) periodically evaluates each endpoint's error rate. If (errorCount429 + errorCount5xx) / totalRequests >= failureRate and totalRequests >= minSampleSize, the breaker transitions to Open. The Tick method uses atomic swaps to read and reset counters simultaneously, preventing races where new requests arrive between reading TotalRequests and ErrorCount429. The sample window is effectively the interval between ticks (default: 5 seconds).
Open (failing): all requests are immediately rejected without contacting the upstream provider, preventing the Gateway from wasting resources on a known-broken endpoint and reducing load on the struggling provider. The breaker stays open for OpenDuration (default: 10 seconds) before transitioning to Half-Open.
Half-Open (probing): a limited number of probes, three by default, go through to test recovery. All succeed and the breaker closes. Any fails and it returns to Open for another OpenDuration. The limit is enforced by tracking hoSuccess + hoFailure rather than by capping concurrent probes, so a burst arriving while half-open lets only the first few through and rejects the rest until the probes resolve. That's intentional — flooding a recovering provider is the fastest way to trigger the next failure cycle.
The Allow method also checks metrics.CircuitOpen in addition to the breaker's own state. This dual-write is a backstop: even if the state machine has a bug — say, failing to transition from Half-Open to Closed correctly — the metrics.CircuitOpen boolean gives external monitoring something to observe and override.
Interplay Between the Layers
- Error Classification determines what kind of failure occurred.
- Retry Strategy determines how to respond to this specific failure — retry, skip, abort.
- Circuit Breaker determines whether to even attempt contacting the provider in the first place.
A typical cascade runs like this. The breaker is closed and requests flow, then the provider starts returning 503s. Each one classifies as ClassTransient, triggers a backoff-retry, and records a failure in the endpoint's metrics. Once minSampleSize requests have accumulated at or above the threshold failure rate, Tick opens the breaker. Subsequent requests are rejected immediately — no more 503s, no more wasted retries. Ten seconds later the breaker half-opens and probes. If the provider has recovered, the probes succeed, the breaker closes, and normal operation resumes.
The system degrades gracefully when a layer fails to help. If the breaker never opens because the failure rate stays below threshold, per-request retry still provides individual resilience. If the retry loop exhausts all keys and endpoints, ErrNoHealthyEndpoint or ErrAllEndpointsDead returns a clean 503 rather than a cryptic internal error.
Why PrepareProposal — Not a Timer Goroutine — Drives Settlement Retries
Cross-chain settlement retries are a special case worth calling out, because the trigger is architecturally different from the API-call retries above. Settlement retries are driven by PrepareProposal, the ABCI method that constructs the next block's transaction set — not by a separate goroutine on a timer. This isn't an implementation convenience, it's a correctness requirement.
Consider the alternative. A background goroutine polls the mainchain for transaction status and resubmits on failure. That races with block production. Suppose the goroutine resubmits a cross-chain transaction while PrepareProposal is building a block containing a conflicting state transition — the user spends the very credits the settlement was supposed to deduct. The resulting block is invalid. Preventing it would take a mutex between the retry goroutine and the consensus reactor, which defeats the point of separating them at all.
By embedding the retry logic in PrepareProposal, ComputeFlux serializes retry decisions through the same consensus pipeline as every other state transition:
PrepareProposalchecks (viaprepareRelayTxs/targetSyncRelay) for pending hub calls (sidechain → mainchain settlement transactions).- If a sync is already in progress (
IsHubSyncRuningreturns true within its 360-second window),PrepareProposalskips user transactions entirely and includes only system transactions — the sync continuation. This prevents ordering conflicts. - If no sync is in progress but hub calls are pending,
PrepareProposalinitiates a new sync via aSysCall_SyncTxStarttransaction. - The sync transaction goes through consensus like any other, with
FinalizeTxexecuting the actual mainchain submission and recording the result. - On failure,
HubSyncEnddeliberately backdates the chain'sLastSynctimestamp so thatIsHubSyncRuningkeeps reporting "busy" for roughly another 60 seconds, then clears — so the nextPrepareProposalthat observesIsHubSyncRuning() == falsere-triggers the same still-pending hub call.
So the cadence comes from that ~60-second cooldown, not from block time — far coarser than the millisecond-scale API backoff above. The risk profiles differ that much. A failed API call costs one user a few seconds of waiting. A failed settlement means a provider goes unpaid for inference work already delivered. Faced with that, ComputeFlux would rather wait out a short mainchain hiccup than hammer the RPC every block.
Cleanup: Signature Data After Retry Exhaustion
When a cross-chain transaction finishes — whether it succeeds or exhausts its retries — the cryptographic artifacts it generated have to be cleaned up. The sync collects DKG partial signatures from validators via sendPartialSignByIds, and those must be deleted afterward so they can't contaminate future transactions. DeleteSigOfTx(txn, txIndex) removes every key-value pair under prefix PartialSigPrefix + txIndex + "_", inside the same database transaction as the block commit.
That atomicity matters. If deletion and commit weren't transactional together, a node crash between them would leave orphaned partial signatures in PebbleDB. On restart, the node might fold those into a new aggregation and produce an invalid threshold signature the mainchain rejects.
Thanks to PebbleDB's LSM-tree structure, the prefix-scan deletion is O(n) in matching keys and O(1) in total database size. With 3–5 validators it's trivially fast. At 100+ it could become a bottleneck, and the fix would be a separate column family or a more granular composite key that supports range deletion.
Cleanup does one more thing. After a successful sync, HubSyncEnd refreshes AsyncBatchState.LastSync and clears the chain's sync state. That's what lets IsHubSyncRuning return false and PrepareProposal resume normal block production.
The UX Trade-Off
Every retry mechanism trades finality against latency. The scheduler's default — three retries starting at 200ms — puts worst-case latency around three seconds before the user sees a failure. For a chat completion API, where responses take 2–10 seconds anyway, that's fine. For a sub-500ms real-time application, it isn't.
Both knobs are per-deployment. A latency-sensitive deployment might run MaxRetries=1, BackoffInitial=50ms, trading success rate for speed. A batch-processing deployment might run MaxRetries=5, BackoffInitial=500ms, trading response time for eventual success.
Cross-chain settlement sits in a different category entirely. Its outcome is binary — paid or not paid — and a failed settlement costs more than a failed inference call. That's why it gets an indefinite, cooldown-gated loop instead of a small bounded retry count.
On-Chain Failure Accounting
Failures aren't just operational noise. They carry economic weight. SettlementWithRoute.FailCount tracks each provider's failures per settlement period, stored in PebbleDB and committed on-chain periodically, which turns them into an auditable SLA record. context.Canceled errors are deliberately left out, so nobody can damage a provider's reputation by connecting and immediately hanging up. What that record enables economically — SLA enforcement, routing reputation, user-side analytics — is Article 10's subject.
Observability as a Compensating Control
The scheduler exposes an OpenTelemetry retryTotal counter tagged by provider, so operators can watch retry rates live. A sudden spike on one provider is a leading indicator of an upstream outage, often visible before that provider's own status page catches up. A gradual rise across all of them suggests broader network degradation.
This is the practical complement to the backoff math above. Those exponential formulas assume failures are independent and randomly distributed. Real failures are frequently correlated — regional outages, provider-wide degradations. Retry-rate monitoring is what lets an operator tell "the math is working as expected" apart from "something is wrong that the math doesn't model."
Key Takeaways
- Four categories, checked in a strict order — and the order is load-bearing. A canceled request can disguise itself as almost any other error. Get the precedence wrong and a user closing their browser tab looks identical to a provider outage, triggering retries that waste real capacity.
- "Temporarily exhausted" and "permanently dead" are tracked separately. A rate-limited credential is set aside for this request and tried again on the next one. A rejected credential is never touched again during this request, no matter how few options remain.
- The randomness in the retry delay is doing real work. Without it, a hundred clients that fail at the same instant retry at the same instant, forever — and the provider's rate limiter never gets a gap wide enough to drain the backlog. A ±20% spread breaks the lockstep within two or three cycles.
- The circuit breaker sits above retries and asks a different question: not how to respond to this failure, but whether to place the call at all. Recovery is tested with a handful of probes, never by reopening the floodgates.
- Money retries differently from requests. A failed inference costs a user a few seconds. A failed settlement means a provider went unpaid for work already performed — so settlement retries indefinitely on a roughly 60-second cooldown instead of quitting after three tries.
- Failures become an auditable on-chain record. With one deliberate exclusion: client disconnects don't count, so nobody can damage a provider's reputation by connecting and immediately hanging up.
Error classification, retry strategy, and circuit breaking together embody a core principle of distributed systems: failures are inevitable, but cascading failures are preventable. By categorizing errors, capping retries, and isolating broken endpoints, ComputeFlux converts unpredictable upstream unreliability into predictable, bounded behavior — clients see either successful responses or clean error codes, never indefinite hangs or retry storms.
That closes out the execution layer. Requests now reach the right machine and survive the wrong one. Part 3 asks the question that turns all of this into a business rather than a demo: who pays whom, how much, and how would anyone prove it?
Next — Article 10: Billing & On-Chain Settlement: the failure counts this article records are one input to a bill. Here's the rest of it — and why a receipt you can verify beats an invoice you have to believe.