§13.4
Chapter 13 · Implementation Guidance and Interoperability Testing

§13.4Security Validation for Handshake and Keying Behavior

RFC 9001RFC 9000

Learning objective. Validate the security-critical behaviors a QUIC/HTTP-3 stack must get right: certificate and ALPN authentication, the anti-amplification and address-validation limits, key-update and AEAD-usage limits, 0-RTT replay protection, and the anti-abuse checks (optimistic ACK, stateless reset) that a hardened implementation enforces.

Security is a set of MUSTs, not a feature #

QUIC folds encryption into the transport (Chapter 2), which means many security properties are transport obligations with MUST-level requirements. Like the framing rules of §13.3, each is a validation target. A stack that completes a handshake and moves data can still be insecure if it skips certificate verification, ignores the amplification limit, or reuses keys past their AEAD limit. This section walks the handshake and steady state as a security checklist.

ServerClientuntil the clientaddress is validated— send ≤ 3× thebytes received(anti-amplification)MUST verify thecertificate chainMUST confirmALPN = h3 — elseclose withno_application_proto-col (0x0178)0-RTT only for idempotent requests ·replay protections MUST beimplementedsteady state — key update before theAEAD confidentiality limit, elseAEAD_LIMIT_REACHEDreject an ACK of an unsent packet →PROTOCOL_VIOLATION · comparereset tokens in constant timeInitial (ClientHello) · padded to ≥1200 bytesInitial + Handshake · certificate ·ALPN "h3"Handshake finished — clientaddress now validated
Fig. 13.4-1The security checks along a connection's life. The client pads its Initial to 1200 bytes; the server holds to the 3× anti-amplification limit until the address is validated; the client MUST verify the certificate and confirm ALPN negotiated "h3"; 0-RTT is replay-limited to idempotent requests; and in steady state the endpoint key-updates before the AEAD confidentiality limit and rejects abuse like an ACK of an unsent packet.RFC 9001 §4.4, §8.1, §6.6; RFC 9000 §8.1

Handshake authentication #

The two checks that make the connection authentic are both the client's ([RFC 9001 §4.4]):

  • Certificate verification. "A client MUST authenticate the identity of the server... verification that the identity of the server is included in a certificate and that the certificate is issued by a trusted entity" ([RFC 9001 §4.4]): the same validation as HTTPS. A stack that accepts an unverified or mismatched certificate has no security at all, and this is the first thing to test (including that it rejects a bad chain).
  • ALPN negotiation. "Endpoints MUST use ALPN" and must confirm they agreed on an application protocol; a failure to select a compatible one is a no_application_protocol alert — "QUIC clients MUST use error 0x0178 to terminate a connection when ALPN negotiation fails" ([RFC 9001 §8.1]). Test that a mismatched or absent ALPN closes the connection rather than proceeding with an assumed protocol.

QUIC also removes a TLS capability: post-handshake client authentication is forbidden. A server "MUST NOT send post-handshake TLS CertificateRequest messages, and clients MUST treat receipt of such messages as a connection error of type PROTOCOL_VIOLATION" ([RFC 9001 §4.4]).

Anti-amplification and address validation #

Before a server has validated a client's address it must not become a reflector. Two limits enforce this (Chapter 8):

  • The 3× limit. "Prior to validating the client address, servers MUST NOT send more than three times as many bytes as the number of bytes they have received" ([RFC 9000 §8.1]), counting all received payload bytes, including discarded ones. A server that exceeds this is a DDoS amplifier; a client's 1200-byte Initial padding ([RFC 9000 §8.1]) is the other half of the mechanism.
  • Token integrity and replay. A Retry or NEW_TOKEN address-validation token "MUST be difficult to guess," "MUST be covered by integrity protection," and servers "MUST ensure that replay of tokens is prevented or limited" ([RFC 9000 §8.1.4]); an invalid token is INVALID_TOKEN ([RFC 9000 §8.1.2]). A stack that issues forgeable or replayable tokens undermines the very validation they provide.

Key update and AEAD limits #

Every AEAD cipher can be used safely only so many times, and QUIC makes the limits explicit ([RFC 9001 §6.6]). An endpoint "MUST count the number of encrypted packets for each set of keys" and "MUST initiate a key update before sending more protected packets than the confidentiality limit for the selected AEAD permits": for AES-GCM that is 2^23 packets. If a key update is impossible or the integrity limit (invalid-packet count) is reached, it "MUST immediately close the connection with a connection error of type AEAD_LIMIT_REACHED" ([RFC 9001 §6.6]). Key updates themselves must be validated too: successfully removing protection with old keys after newer keys were used at lower packet numbers is a KEY_UPDATE_ERROR ([RFC 9001 §6.4]). These are easy to overlook because they only trigger on very long-lived or heavily-attacked connections, which is exactly why they need deliberate testing rather than waiting to hit them in production.

0-RTT replay #

0-RTT trades a round trip for replay exposure (§2.3), and the obligation is firm: "Endpoints MUST implement and use the replay protections described in [TLS13]," and "the responsibility for managing the risks of replay attacks with 0-RTT lies with an application protocol" ([RFC 9001 §9.2]). For HTTP/3 this means the anti-replay handling of [RFC 8470] MUST be applied ([RFC 9114 §10.9]): in practice, restricting 0-RTT to idempotent requests (§8.3). Validate that non-idempotent requests are not executed from 0-RTT, and that the server can reject 0-RTT and have the client safely retry at 1-RTT.

Anti-abuse checks worth testing #

A hardened stack also defends against peers that lie:

  • Optimistic ACK. An endpoint "SHOULD treat receipt of an acknowledgment for a packet it did not send as a connection error of type PROTOCOL_VIOLATION" ([RFC 9000 §13.1]); a common technique is to skip packet numbers deliberately to detect a peer that ACKs them anyway ([RFC 9000 §21.4]): a peer forging ACKs to inflate the congestion window.
  • Stateless reset comparison. When matching a datagram's trailing bytes against known stateless reset tokens, an endpoint "MUST perform the comparison without leaking information about the value of the token... in constant time" ([RFC 9000 §10.3.1]); a naïve byte-by-byte compare is a timing side channel.

Worked example: validating a long-lived connection #

A test opens a connection and drives it far past normal — billions of packets — to exercise the keying path. A correct stack initiates a key update well before 2^23 packets under one AES-GCM key, toggling the Key Phase bit (§2.2); the harness confirms the update happens and that data keeps flowing across it. Then it forces the pathological case (key update disabled) and confirms the stack closes with AEAD_LIMIT_REACHED rather than continuing to encrypt past the confidentiality limit. In the same run it injects an ACK for a packet number the client skipped and confirms a PROTOCOL_VIOLATION close. None of these paths is reached by ordinary traffic; all are reached by an attacker or a very busy production connection, which is why they are on the security test plan and not left to chance.

Note

The uncomfortable truth about these checks is that skipping them is invisible until exploited. A stack that never verifies certificates interops perfectly with honest peers; one that ignores the AEAD limit runs fine for months; one with a timing side channel in its reset comparison passes every functional test. Unlike a framing bug, a missing security check does not announce itself: the connection works, right up until an adversary makes it not work. That is why security validation must be an explicit, adversarial test plan, not an assumption that "it connected, so it's fine."

In practice

editorial Build a security test plan that an attacker's checklist would recognize: present a bad certificate and assert rejection; force ALPN mismatch and assert 0x0178; verify the server never exceeds 3× before validation; drive a connection past the AEAD key limit and assert a key update (and AEAD_LIMIT_REACHED when updates are blocked); replay a 0-RTT request and assert it is not double-executed; and forge an ACK for an unsent packet and assert PROTOCOL_VIOLATION. Lean on your TLS library for certificate and key-schedule correctness rather than reimplementing it, but test that you actually call it correctly: the most common real failure is not a broken crypto primitive but a stack that forgot to check the primitive's result.

Takeaways #

QUIC's security is a set of transport MUSTs to validate, not a feature to assume: the client must verify the certificate and confirm the h3 ALPN (no_application_protocol/0x0178 on failure); the server must honour the 3× anti-amplification limit and issue unforgeable, replay-resistant tokens; endpoints must key-update before the AEAD confidentiality limit (AEAD_LIMIT_REACHED otherwise) and restrict 0-RTT to idempotent requests; and a hardened stack rejects optimistic ACKs and compares reset tokens in constant time. Because missing security checks are invisible until exploited, they demand an explicit adversarial test plan, which §13.5 assembles, with everything else, into a production-readiness checklist.