In one sentence. When an AI answer arrives one word at a time, ComputeFlux has to convert it into a different format on the fly — never waiting for the sentence to finish.
Picture it like this. A live interpreter who must start speaking before the speaker has finished, into a language that orders its grammar differently. They can't wait for the complete thought; they hold fragments in their head and commit to a rendering as they go, occasionally discovering the sentence went somewhere they didn't expect. That's why streaming translation is strictly harder than the batch translation in Article 12.
Why it matters. The typewriter effect you see in every AI chat product is the single largest factor in how fast it feels. It's also where the most fragile plumbing hides: mishandle a chunk boundary and text garbles silently; mishandle backpressure and one user on a slow connection can starve the server's memory.
The genuinely hard part is the tool-call state machine — skip there if you're short on time.
Streaming LLM responses pose a protocol design challenge unlike any other web payload. Follow one through the system. The provider emits tokens at 20–50 per second. They cross the public internet, reach ComputeFlux's Gateway, and go through protocol translation between OpenAI, Anthropic, and Gemini formats — the problem Article 12 introduced and deferred here. Then each token has to land in the user's browser with no perceptible added latency.
Two things make that hard. Chunk boundaries carry no semantic meaning, so a single JSON object may arrive split across two TCP segments. And compound constructs like tool calls require state maintained across chunks. Get either wrong and you get silent data corruption — garbled text — or a hung stream that eventually times out. This article is the canonical deep-dive on that pipeline.
SSE vs WebSocket: Why One-Way Events Win for LLM Streaming
The first architectural decision is the transport protocol. Both Server-Sent Events (SSE) and WebSocket can deliver server-to-client streaming, but they make fundamentally different assumptions about connection lifecycle and bidirectional capability.
SSE is a unidirectional protocol layered on HTTP. The client opens a standard HTTP GET request with Accept: text/event-stream, and the server responds with Content-Type: text/event-stream followed by a persistent connection over which it writes data: lines. There are no protocol-level heartbeats, no ping/pong frames, no upgrade handshake beyond the initial HTTP request. SSE inherits all of HTTP's infrastructure: proxy support, HTTP/2 multiplexing, Content-Encoding for compression, standard load-balancer health checks. For a gateway that terminates at HTTP load balancers and reverse proxies, SSE requires zero infrastructure changes.
WebSocket is bidirectional, beginning as an HTTP Upgrade request and, after a 101 Switching Protocols response, operating as a full-duplex binary or text frame protocol. That bidirectional capability is powerful but irrelevant for LLM streaming: the client sends one prompt and receives one stream of tokens, with no back-and-forth dialogue within a single generation. The bidirectional capability becomes dead weight — frame masking, ping/pong keepalive, reconnection semantics — without providing value.
SSE's simplicity also makes it auditable. An SSE stream is human-readable plaintext (data: {"choices":[{"delta":{"content":"Hello"}}]}\n\n); a WebSocket stream is binary frames that need a protocol analyzer to inspect. For a system whose core value proposition is TEE-based trust, having an inspectable wire format matters: a user can run curl -N against the gateway and see exactly what data is flowing, with no specialized tools. This transparency extends to debugging provider issues — an operator can inspect a provider's raw SSE output by piping it through the gateway's debug endpoint, which logs raw chunks before conversion.
The Chunking Problem: Token Boundaries vs. Network Packet Boundaries
LLMs produce output token-by-token, each token typically 1–4 characters of text and 3–50 bytes JSON-encoded (e.g. {"choices":[{"delta":{"content":" the"}}]}). The network operates in packets of up to 1500 bytes (Ethernet MTU) or ~1400 bytes (typical TCP MSS after IP/TCP headers). At 30 tokens/second, a provider produces roughly 450–1500 bytes/second — about one network packet's worth.
The mismatch between token boundaries and packet boundaries means the gateway can't assume each TCP Read() returns exactly one complete JSON object. A single Read() might contain half of chunk N's JSON, all of chunk N+1, and the beginning of chunk N+2 — or exactly 1.2 chunks. The gateway has to reassemble a byte stream into a sequence of discrete SSE events, each a syntactically complete JSON object.
ComputeFlux's chunkReader handles this by implementing io.Reader over the scheduler's Stream.Next(ctx) method. Next() returns discrete chunks as produced by the provider's HTTP client (which itself buffers and reassembles TCP segments). chunkReader maintains an internal []byte buffer accumulating partial chunks: when Read(p) is called with an empty buffer, it fetches another chunk from the scheduler; if the fetched data is larger than p, the excess stays buffered for the next call. This is standard io.Reader buffering — the innovation isn't the algorithm but the lifecycle integration with the scheduler's streaming abstraction. The scheduler's Stream interface abstracts over providers with different wire formats — for an OpenAI-compatible provider, Next() returns raw SSE lines (data: {...}\n\n); for a gRPC-streaming provider, it unpacks protobuf frames into the same byte-slice format. chunkReader is format-agnostic, seeing only []byte chunks from an abstract stream; the protocol conversion layer (streamConvert) interprets the bytes according to the provider's expected format.
The Backpressure Problem: When the Client Is Slow and the Provider Is Fast
Backpressure arises when tokens arrive from the provider faster than the client can consume them — common when a provider like Groq generates 500+ tokens/second but the end user is on a 3G connection draining the HTTP response buffer at 50KB/s. Without backpressure, the gateway's memory buffer grows unboundedly, eventually triggering an OOM kill or, in the TEE context, exhausting EPC memory — which triggers a cascade of EPC page faults degrading all concurrent streams.
ComputeFlux's backpressure mechanism comes from Go's HTTP ResponseWriter semantics. When the gateway calls ResponseWriter.Write(), Go's HTTP server buffers it and flushes to the TCP socket when the buffer fills, the handler returns, or Flusher.Flush() fires.
Now suppose the client stops reading. The socket's send buffer fills, and Write() blocks. That block then propagates backward through the whole pipeline:
flushWriter.Write() blocks → streamConvert can't write its output → streamConvert stops reading from chunkReader → chunkReader stops calling s.Next() → the scheduler stops pulling from the provider's HTTP response body → the provider's TCP receive window closes → the provider stops generating tokens, or buffers them itself.
This is TCP backpressure, and it works correctly for most scenarios, but it has a critical failure mode: intermediate buffers. Go's HTTP server bufio.Writer holds 4KB before flushing; the OS TCP send buffer holds 16KB–64KB (configurable via sysctl net.core.wmem_default); the provider itself may buffer several seconds of tokens. The total elasticity — data that can be in-flight before the provider actually pauses generation — is typically 100–500KB, representing 10–60 seconds of generation at typical speeds. If the client stays slow longer than that elasticity window, the buffer fills and the gateway process experiences sustained write blocking.
ComputeFlux doesn't put a short per-write deadline on the ResponseWriter. That would risk cutting off a client that's slow but perfectly alive. Instead, before the first byte of a stream goes out, the handler calls http.NewResponseController(w).SetWriteDeadline to clear the connection's write deadline entirely. The stream's lifetime then belongs to the request context, not to a fixed per-write timer. Where that override isn't supported, the server's global WriteTimeout — covered below — serves as a backstop.
A genuinely stuck write, where the client has vanished and the TCP send buffer is permanently full, gets cleaned up when the request's context is canceled: by the client disconnecting, by the gateway's request-level timeout, or in the worst case by that WriteTimeout backstop.
Why the Reader/Converter Boundary Doesn't Need Its Own Buffer Cap
A separate buffer problem could in principle appear inside the gateway, between the stream reader and the protocol converter, if the converter ran slower than the provider. Anthropic-to-OpenAI conversion is a plausible candidate, since it reconstructs JSON objects that Anthropic delivers as structured SSE events.
It doesn't happen, because of how chunkReader is built. It's a synchronous io.Reader. Its Read() only fetches a new chunk from the underlying scheduler.Stream after the caller has fully drained the internal buffer, and it holds at most one recently-fetched chunk at a time — cr.buf = chunk.Data, consumed through Read() until empty before the next s.Next() call.
That makes it pull-based and self-limiting by construction. It can never run more than one provider-emitted chunk ahead of its consumer, so no unbounded backlog can accumulate between reader and converter. A slow converter just means chunkReader calls s.Next() less often — which is the same TCP-backpressure chain from above, propagating back to the provider.
So ComputeFlux needs no bounded ring buffer or token-dropping mechanism at this layer, and the current code implements neither. Memory stays bounded through the backpressure already described — blocking Write() calls travelling back through chunkReader to the scheduler's stream and finally to the provider connection — not through an independent cap-and-drop scheme.
The Tool Call State Machine Across Stream Chunks
Tool calls (function calling) are the hardest streaming construct to translate between protocols, because they aren't atomic. OpenAI delivers a tool call across multiple sequential chunks: first a chunk with id and function.name, then one or more chunks with fragments of function.arguments, finally a chunk with finish_reason: "tool_calls". The arguments arrive as incremental JSON string fragments that must be concatenated into a valid JSON object before the tool can be invoked. Anthropic delivers tool use as a content_block_start event with the full tool name and ID, followed by content_block_delta events with partial JSON, then a content_block_stop event.
Converting between these means holding per-tool-call state across chunks. The converter keeps a map from tool_call_index (OpenAI) or content_block_index (Anthropic) to a partial state object carrying the accumulated name and arguments. Each incoming chunk updates the state at its index. When the call completes — signaled by finish_reason or content_block_stop — the converter emits the representation the target protocol expects.
Three edge cases keep that state machine honest.
A provider may interleave text deltas and tool-call deltas within OpenAI's stream. The converter has to track whether the current output mode is text or tool_call and switch cleanly between them.
A provider may send tool-call chunks out of index order — index 1 before index 0, which OpenAI's protocol permits. So the converter must be able to initialize state for index 1 before index 0 has ever appeared.
A provider may cancel a tool call mid-stream, sending a content_block_stop with no finish_reason: "tool_calls". The converter then has to discard the partial arguments it accumulated for that call.
Anthropic's tool-use format adds a wrinkle: its content_block_start event includes the tool's JSON Schema, defining expected parameters, while OpenAI's format includes only the function name and arguments (the schema comes from the client's initial request). Converting Anthropic to OpenAI, the converter strips the JSON Schema from the stream — irrelevant to an OpenAI client that already has the schema — emitting only name and accumulated arguments. Converting OpenAI to Anthropic, the converter has to synthesize a content_block_start event with a reconstructed tool name and empty arguments, then emit content_block_delta events as argument fragments arrive.
The 600-Second Timeout: Why 10 Minutes
The streaming timeout is set to 600 seconds, which looks extravagant for most LLM requests that complete in 3–30 seconds. The rationale is specific to reasoning models: OpenAI's o1-pro performs internal chain-of-thought reasoning before emitting any visible tokens, and for complex prompts — multi-step mathematical proofs, legal document analysis, codebase-wide refactoring plans — this reasoning phase can take 3–8 minutes. During this entire period, the SSE connection stays open but silent: no data: events, no heartbeats (SSE has no built-in keepalive mechanism). The client sees nothing until reasoning completes.
At a typical 60-second HTTP timeout, every non-trivial o1-pro request would time out before the first token appeared. The 600-second value is the observed upper bound for o1-pro on the most complex reasoning benchmarks (GPQA, MATH-500) plus a 50% safety margin — not a target (most streams complete in under 30 seconds), but a ceiling preventing premature termination.
The 600-second figure appears as the HTTP server's WriteTimeout, and it's easy to misread. It does not reset on every successful write. Go's http.Server.WriteTimeout is a deadline anchored to the connection and request, not something a handler pushes forward each time it calls Write().
That's exactly why ComputeFlux can't lean on it for long streams. Before writing the first byte, the handler clears the per-request write deadline with http.NewResponseController(w).SetWriteDeadline(time.Time{}) and hands the stream's lifetime to the request context. The 600-second WriteTimeout stays as a fallback for the rare connection where that override isn't supported — and there, a stream is bounded by the full 600 seconds from when the response started, not by any per-token idle window.
The 600-second value also touches billing, which settles every 5 minutes (Article 10). Usage is recorded only once a stream finishes and its final token counts are known. A stream that outlives one or more settlement ticks simply gets picked up by whichever cycle runs after it completes. The 600-second cap guarantees that even the longest stream terminates and gets recorded eventually, so no indefinitely-open connection leaves its billing data permanently in flight.
Key Takeaways
- The simpler transport wins. One-way streaming over ordinary HTTP inherits proxies, load balancers, and compression for free, while a full bidirectional socket adds machinery for a conversation that never actually happens. It's also human-readable — which counts for something in a system whose whole pitch is verifiability.
- The network doesn't respect word boundaries. A single read might contain half of one message, all of the next, and the opening of a third. Reassembling that correctly is unglamorous work and the most common source of silent text corruption in streaming gateways.
- Backpressure is inherited rather than invented. When the reader is slow, writes block, and that blocking propagates all the way back until the provider itself stops generating. No separate buffer was added, deliberately: the reader can never get more than one chunk ahead of its consumer.
- Tool calls are the genuinely hard part, because they aren't atomic. They arrive in fragments spread across many messages, in three incompatible shapes, sometimes out of order, and occasionally canceled halfway through.
- Ten minutes sounds absurd for a chat request — until you meet reasoning models that think silently for eight minutes before emitting a single visible word. The ceiling exists for them, not for the common case.
Streaming completes the picture of how a request gets in and an answer gets out. What's still missing is everything around the request: accounts, keys, balances, and the dashboard a person actually clicks on.
Next — Article 15: GraphQL API Layer: why ComputeFlux runs two completely different API styles, and why that's the correct answer rather than a compromise.