In one sentence. An AI model briefly remembers the work it did on your last prompt, so ComputeFlux tries hard to send your next request to the exact same machine.
Picture it like this. A chef who has already prepped your ingredients. Order from the same kitchen within a few minutes and the dish arrives almost instantly; order from a different kitchen and they start from a cold station. Conventional load balancers spread work evenly across kitchens — which, here, is precisely the wrong instinct.
Why it matters. This one behavior is the difference between waiting 3–8 seconds and waiting half a second, at roughly half the price. It isn't a micro-optimization; it's one of the largest cost levers in the entire gateway.
Know why sticky sessions matter? Skip to the latency scoring.
LLM inference has paradoxical economics: repeating a computation costs almost nothing, while doing it the first time costs a great deal. ComputeFlux's scheduler exists to exploit that asymmetry. It does so across a routing table that spans everything from OpenAI-proxied endpoints to personal GPUs behind residential NAT — the two ends described in Article 6 and Article 7.
When you send a prompt to an LLM, the inference engine — vLLM, TGI, and the like — caches the intermediate representations of the input tokens, the Key-Value pairs, in GPU VRAM. Send that same prompt again to the same GPU instance and the engine skips the entire prefill phase and jumps straight to decode. That's a KV Cache hit, and the difference is dramatic. Take GPT-4o with a 128K context:
| Scenario | Prefill Latency | Decode Latency | Total Latency | Cost |
|---|---|---|---|---|
| KV Cache MISS | 3-8 seconds | 0.5 seconds | 3.5-8.5 seconds | Full price |
| KV Cache HIT | 0 seconds | 0.5 seconds | 0.5 seconds | 50%+ discount |
But the fatal limitation of KV Cache is that it's bound to a specific GPU instance. Round-robin load balancing across endpoints drives the KV Cache hit rate toward 0%. This is why Session Affinity exists in ComputeFlux's scheduler — not because sticky sessions are inherently desirable, but because of the physical constraints of KV Cache.
Session Affinity: Remembering the Last Successful Endpoint
ComputeFlux implements Session Affinity as a small TTL-based cache, not a deterministic hash. It's a mutex-protected map from session ID to the endpoint key that last served that session successfully, and each entry expires after a fixed TTL — five minutes by default — once the session goes idle. A request looks up its session ID. A live entry means "try this endpoint first," and a successful response refreshes the mapping.
That's a different trade-off from hashing (userID, modelName) statelessly. The appeal of a hash is that it needs no shared state: every node computes the same answer independently. Its weakness is that it can't adapt. When the chosen endpoint disappears or degrades, the request keeps hashing to the same bad choice until the input itself changes.
A last-successful-endpoint cache adapts naturally. It forgets a mapping once nothing has touched it for the TTL window, and a caller can evict an entry outright after a hard failure instead of being stuck with a fixed outcome. The cost is what adding state always costs: the cache lives on whichever process handled the previous request, so it doesn't generalize across independent scheduler instances the way a pure function of the input would.
EWMA Latency Prediction: Why Exponential Weighting Instead of Simple Averaging
Endpoint latency is the most important signal when selecting an endpoint, but "average latency" is a deceptive metric. Suppose an endpoint's latency sequence is [100ms, 200ms, 100ms, 200ms, 10000ms, 100ms] — where the last measurement was an accidental timeout. Simple average: ~1783ms (10700 ÷ 6), completely polluted by that single outlier, misleading the next 100 requests.
EWMA (Exponentially Weighted Moving Average) solves this:
EMA_new = (1 - α) × EMA_old + α × current_latency
At α = 0.3, each new measurement moves only 30% of the weighted value. The 10000ms outlier contributes 3000ms on its first appearance (0.3 × 10000), pushing the EMA to roughly 3.1s before it decays by a factor of 0.7 on each subsequent update — dropping below the polluted simple average within a couple of normal measurements. That's the core insight: EWMA forgets outliers, and the forgetting is the point. Transient jitter shouldn't permanently damage an endpoint's score.
Choosing α is a real design decision. Push it higher and the score reacts fast but swings wildly on occasional anomalies. Push it lower and the score smooths out but takes longer to flag genuine degradation. ComputeFlux defaults to 0.3, which balances the two.
The Cold Start Problem and coldStartSamples
EWMA needs an initial value. If an endpoint has just come online with no historical data, what should its score be? The simplest approach — an "optimistic initial value" assuming the new endpoint is fast — causes traffic to instantly flood it, potentially overwhelming it. This is the cold start problem: a new endpoint has neither failure records (looks fine) nor latency data (can't judge speed).
ComputeFlux's answer is to withhold judgment. While an endpoint's total request count sits below the cold-start threshold (coldStartThreshold), the scoring function skips the success-rate, load-factor, and latency terms entirely and returns the endpoint's static configured weight. It competes on declared priority alone until it has built enough of a track record for the real-time signals to mean anything.
if total < coldStartThreshold {
return weight // cold start: score purely on static weight
}
// otherwise: weight × successRate × loadFactor / latencyFactor
There's a subtlety here. An endpoint that fails from its very first request would, under weight-only scoring, keep receiving traffic through the whole cold-start window and keep failing. What bounds the damage is the window itself: ColdStartSamples defaults to 20, so an unproven endpoint gets a limited number of chances, not unlimited ones. Fail through all of them and the total request count crosses the threshold, the success-rate term switches on, and the score drops accordingly.
Half-Open Probing in Circuit Breakers
The standard circuit breaker is Netflix Hystrix's three-state model, and the AI gateway breaks one of its assumptions: 429 rate-limiting and 503 overload recover in completely different ways. A recovered 503 can take full traffic immediately. A 429 will reject everything new until the endpoint's rate-limit window — usually a minute — expires. Apply the textbook "half-open, then admit all" strategy to a 429 and the breaker trips again instantly, oscillating.
So ComputeFlux's half-open state admits a small, bounded number of probes instead of opening the floodgates. HalfOpenProbes defaults to 3. If they succeed, the endpoint has genuinely recovered — the rate-limit window passed, or the load dropped. If they fail, it returns to OPEN and the cooldown timer resets.
That trades recovery speed for stability: a handful of probes per attempt rather than full traffic, in exchange for never re-tripping the breaker before the rate-limit window closes. For an AI API gateway, safety over speed is the right call. Article 9 picks up directly from here with the full mechanics — error classification, retry and backoff, and how they interact with this probing.
Priority Ordering of Endpoint Exclusion Strategies
The Scheduler excludes four categories of endpoints when making a selection, in priority order:
- Disabled (permanent exclusion): endpoints manually taken offline by the provider — highest priority to skip.
- DeadKeys (401/403 authentication failure): the key itself is invalid, marked immediately, no retry within this request — though it may be retried across different requests, since temporary credentials can become valid again after refresh.
- UsedKeys (429 rate-limit already attempted): tried within this request and rate-limited, skip — different requests will retry.
- Model Mismatch: the endpoint simply doesn't contain the requested model — permanent skip.
The ordering keeps DeadKeys and UsedKeys scoped to a single request, so neither pollutes endpoint state across requests. Cross-request state lives elsewhere entirely — in the circuit breaker and the EMA latency score described above.
Key Takeaways
- The economics are lopsided, and everything here exploits that. Repeating work a machine has already done is nearly free; doing it the first time is expensive. A scheduler that ignores this leaves most of the available savings on the table.
- Even distribution is the enemy. Textbook round-robin balancing drives the cache hit rate toward zero. The "fair" answer is the expensive one.
- Averages lie, so the scheduler uses one that forgets. A single ten-second timeout poisons a plain average for the next hundred requests. An exponentially weighted average lets outliers decay — forgetting is the feature, not a bug.
- A brand-new endpoint is a genuine dilemma. It has no failures on record (so it looks perfect) and no speed history (so it can't be judged). It competes on declared priority alone for a bounded number of requests, then the real numbers take over.
- Recovering endpoints get a few probes, not the firehose. Hand full traffic back to a rate-limited provider and it simply trips again — the standard textbook behavior would oscillate forever.
Session affinity, EWMA scoring, and cold-start handling together make the Sticky Balancer a scheduler that optimizes for KV cache economics without sacrificing resilience — but scheduling and failure handling are two sides of the same coin.
Next — Article 9: Resilience — Circuit Breakers & Retry/Backoff: picking a good endpoint is half the job. The other half is what happens at 3am when that endpoint stops answering.