§9.6Streaming LLM Inference Responses (SSE and Streamable HTTP) over HTTP/3

Learning objective. See how token-by-token LLM inference streaming, usually delivered as Server-Sent Events (SSE) or a plain streamable HTTP body, maps onto HTTP/3, why it is a reliable real-time workload rather than a datagram one, and what QUIC specifically buys it: head-of-line isolation, faster first token, clean cancellation, and survival across network changes.

A different kind of real-time #

The rest of this chapter argued that real-time media wants unreliable delivery, because a late video frame is useless (§9.3). LLM inference streaming looks superficially similar (a low-latency, incremental stream a human watches appear in real time) but it inverts the key property: every token must arrive, in order. Drop token 5 of "The capital of France is Paris" and the output is corrupted; reorder them and it is gibberish. Late data is still entirely useful; you simply cannot skip it. So token streaming is a reliable, ordered real-time workload, and the right transport model is a reliable QUIC stream (§9.3), never a datagram. It is the instructive counter-case: real-time does not always mean loss-tolerant.

That reframes which chapter tools apply. Datagrams, FEC, and deadline-dropping are off the table. What matters instead is time to first token, inter-token latency and its jitter, isolation of concurrent requests, and keeping a long generation alive, all of which HTTP/3 addresses directly.

SSE and streamable HTTP are just DATA frames #

Most inference endpoints stream with one of two HTTP mechanisms, and over HTTP/3 they converge to the same thing on the wire:

• Server-Sent Events (SSE). The response carries content-type: text/event-stream, and the body is a sequence of data: … events, one per token or token-group. SSE is a web standard defined at the HTTP-message level, so it rides any HTTP version unchanged.
• Streamable HTTP. A plain response body (often newline-delimited JSON) flushed incrementally as the model generates. No special media type, just bytes sent over time.

Both are, in HTTP/3 terms, identical: a single HEADERS frame for the response status, then a series of DATA frames carrying the body as it is produced, then the stream ends ([RFC 9114 §4.1]). Crucially, HTTP/3 has no chunked transfer-encoding: "the Transfer-Encoding header field MUST NOT be used" ([RFC 9114 §4.1]). It does not need one, because the DATA-frame sequence is the incremental framing, and the QUIC stream's FIN delimits the end. Where HTTP/1.1 bolts streaming on with Transfer-Encoding: chunked, HTTP/3 streams natively.

Seen as a flow over time, one generation is a HEADERS frame followed by a DATA frame per token, flushed as each is decoded, until the stream closes:

What HTTP/3 buys token streaming #

Running the same SSE/streaming response over QUIC instead of TCP changes four things that matter for inference serving:

1. No head-of-line blocking across concurrent generations. A client running many inference calls at once (an agent fanning out sub-tasks, a batch job, a UI with several panels) multiplexes them over one connection. Over HTTP/2 on TCP, a single lost packet stalls every multiplexed stream until it is retransmitted (§4.4); one slow network moment freezes all the token streams together. Over HTTP/3 each request is an independent QUIC stream, so a loss affecting one generation leaves the others flowing (§4.4). For concurrent LLM traffic this is the headline win.

2. Faster time to first token. The first token is the latency users feel most. A cold connection spends a round trip on the handshake before the request even arrives (§8.1); a warm, pooled connection removes it, and 0-RTT (§8.3) can carry the request in the first flight, provided the request is safe to replay (§2.3), which a stateless completion call typically is. Keeping connections warm to an inference endpoint is the cheapest first-token win available.

3. Clean, cheap cancellation. Users stop generations constantly: a wrong turn, a good-enough answer. On QUIC the client sends RESET_STREAM (and STOP_SENDING) to abort that one stream (§5.4), which tells the server to stop generating and frees expensive GPU compute immediately, without touching other in-flight requests on the connection. Cancellation is a first-class, per-stream operation rather than a connection teardown.

4. Survival across network changes. A long generation can run for tens of seconds. A mobile client that switches from Wi-Fi to cellular mid-response would, on TCP, drop the connection and lose the partial output. QUIC's connection migration (Chapter 10) keeps the same connection — and the same streaming response — alive across the path change, so the tokens keep coming.

The within-stream latency still applies #

One caveat keeps the picture honest. Cross-stream isolation does not remove head-of-line blocking within a single generation's stream: because tokens are delivered reliably and in order, a lost packet still delays the tokens behind it on that stream by about a round trip (§9.1). This is unavoidable (the tokens must be ordered), so the mitigations are the general ones: a low-RTT path, pacing to avoid self-inflicted loss (§9.2), and prompt flushing. What HTTP/3 removes is the cross-request amplification of that stall, not the intra-stream cost of ordered delivery.

Two operational points follow from earlier sections:

• Flush every token; don't batch. The server must emit each token (or small group) as a DATA frame as it is generated, not accumulate them. Write coalescing or Nagle-style batching directly inflates inter-token latency, the very thing streaming exists to minimize (§9.4).
• Flow-control backpressure is a feature here. For media, flow control's backpressure was harmful (§9.4); for reliable token streaming it is welcome. If the client reads slowly, the stream's flow-control window fills and naturally paces the server, so it does not generate far ahead of consumption and buffer unbounded output. The same mechanism that hurt datagram media helps reliable streaming.

Worked example: a chat completion on a warm connection #

Trace Figure 9.6-2 for a chat UI. The client holds a warm, pooled HTTP/3 connection to the endpoint, so there is no handshake round trip; it sends the completion request as a single HEADERS frame (0-RTT would even fold it into the first flight on a fresh connection). The server responds with a HEADERS frame carrying 200 and text/event-stream, then streams a DATA frame per token as the model decodes (The, capital, of, …), each flushed immediately, arriving in order and reliably. When the user clicks stop, the client resets the stream; the server halts decoding and reclaims the GPU, while a second generation open on another stream of the same connection streams on undisturbed. Had a packet been lost, only this generation's subsequent tokens would have waited a round trip; the other stream would not have noticed.

Takeaways #

LLM token streaming is a reliable, ordered real-time workload — the counter-case to media — so it uses a reliable QUIC stream, not datagrams. SSE and streamable HTTP both reduce over HTTP/3 to a HEADERS frame plus a sequence of DATA frames flushed per token, with no chunked encoding needed. QUIC's gains are cross-stream head-of-line isolation for concurrent generations, faster first token via warm connections and 0-RTT, clean per-stream cancellation, and connection migration for long responses. Ordered delivery's within-stream cost remains, managed by prompt flushing and welcome flow-control backpressure. That closes the real-time chapter; the chapter quiz checks the model, and Chapter 10 turns to the migration that keeps these long-lived streams alive as the network path changes.