Stream reassembly memory-exhaustion guard in quinn-proto can be bypassed

The fix for RUSTSEC-2026-0185 in 0.11.15 added a cap on the number of chunks buffered by the Assembler that reassembles out-of-order stream data. However, the cap was only checked inside the branch that triggers defragmentation, which only runs when the estimated over-allocation exceeds a threshold proportional to the amount of buffered data. Frames handled entirely by the unordered-read deduplication loop returned before the check ran at all.

A peer sending one small gapped STREAM frame per packet (for example, 6-byte frames separated by 1-byte gaps) keeps the per-chunk overhead below the defragmentation threshold, so the cap is never evaluated and the chunk count grows until stream-level flow control stops the peer. With the default TransportConfig (100 concurrent unidirectional streams and a 1.25 MB stream receive window), a remote, unauthenticated peer can pin roughly 3 GB of receiver memory using about 125 MB of accounted stream data. The memory is not reclaimed until the affected streams or the connection are closed.

The flaw was corrected in 0.11.17 by checking the chunk count after every insert (including in the unordered deduplication loop) and compacting the buffer once it passes a watermark above the cap. A stream is rejected once compaction can no longer bring the chunk count back under the cap.

Versions before 0.11.15 are affected by the original, broader issue described in RUSTSEC-2026-0185.

Thanks to Otto Hermann (@ojhermann) for reporting and fixing this issue.