REDHAT-BUG-2536932: High severity Netty maven/io.netty/netty-codec-http vulnerability

Unbounded Per-Connection Queue Growth in WebSocketServerExtensionHandler Leads to Denial of Service

A public GitHub Security Advisory (GHSA-2g37-3h88-55hc) describes the following issue:

Summary

WebSocketServerExtensionHandler keeps a per-channel Queue<List<WebSocketServerExtension>> field named validExtensions. It offers one entry for every inbound HttpRequest and polls one entry only when the application later writes an HttpResponse. Nothing bounds the queue. A remote, unauthenticated peer that uses HTTP/1.1 pipelining to send requests faster than the application produces responses grows the queue without limit until the JVM exhausts heap and dies with OutOfMemoryError.

The affected handler is the base class of WebSocketServerCompressionHandler, the standard handler applications add to enable permessage-deflate. Any server that supports WebSocket compression is exposed on its plain HTTP port, before any WebSocket upgrade completes and before any authentication the application may perform.

Details

The queue is declared as per-connection state in codec-http/src/main/java/io/netty/handler/codec/http/websocketx/extensions/WebSocketServerExtensionHandler.java:

java private final Queue<List<WebSocketServerExtension>> validExtensions = new ArrayDeque<>(4);

It is filled on the inbound path, once per request, with no size check of any kind:

java protected void onHttpRequestChannelRead(ChannelHandlerContext ctx, HttpRequest request) throws Exception { List<WebSocketServerExtension> validExtensionsList = null; ... if (validExtensionsList == null) { validExtensionsList = Collections.emptyList(); } validExtensions.offer(validExtensionsList); // unbounded super.channelRead(ctx, request); }

It is drained only on the outbound path, once per response:

java protected void onHttpResponseWrite(ChannelHandlerContext ctx, HttpResponse response, ChannelPromise promise) throws Exception { List<WebSocketServerExtension> validExtensionsList = validExtensions.poll(); ... }

Three properties make this remotely drivable rather than merely untidy.

The fill rate is under the attacker's direct control and the drain rate is not. onHttpRequestChannelRead runs on the I/O thread the instant bytes are decoded, whereas onHttpResponseWrite runs only when the application chooses to write a response. Any application that does asynchronous work before responding, which is the common case for a WebSocket endpoint that consults a database or an auth service, drains strictly slower than an attacker can fill.

The offer happens for every HttpRequest, not only for WebSocket upgrades. When the request is not an upgrade the handler still offers, using Collections.emptyList(). Plain GET requests to any path therefore grow the queue.

The handler removes itself from the pipeline only after a successful 101 Switching Protocols response. A peer that never completes an upgrade keeps the handler, and its queue, alive for the whole life of the connection.

Supplying a valid Sec-WebSocket-Extensions: permessage-deflate offer makes each entry a real ArrayList holding a PerMessageDeflateServerExtension instance rather than the shared empty-list singleton, which is what raises the per-entry cost from a queue slot to roughly 100 bytes.

Proof of concept

PocPipelineQueue.java in this report drives the released io.netty:netty-codec-http:4.2.17.Final artifact through EmbeddedChannel and reads the queue depth reflectively. It runs three stages. Build and run with the attached pom.xml:

mvn -q -B compile exec:java

Observed output:

[websocket] WebSocketServerCompressionHandler requests in : 200000 responses written : 0 validExtensions depth : 200000 retained heap growth : 17.9 MiB (94 bytes/request) bound enforced : NO

[control] HttpContentCompressor requests accepted : 128 refused with : IllegalStateException: maxPipelineDepth exceeded: 128 bound enforced : yes

[raw wire] HttpServerCodec + WebSocketServerCompressionHandler pipelined requests : 50000 bytes sent on the wire : 9750000 (195 bytes/request) validExtensions depth : 50000 retained heap growth : 4.8 MiB reachable from network : YES - decoded HTTP bytes alone drive the queue

The first stage shows the queue growing one-to-one with inbound requests and never stopping.

The second stage is a control. HttpContentCompressor maintains a structurally identical per-connection queue and already carries a depth bound, so running the same harness against it shows what a bounded handler does: it refuses the 129th request. Because the control refuses and the WebSocket handler does not, the difference is the missing bound in the WebSocket handler and not an artifact of how the harness feeds requests.

The third stage removes any doubt about network reachability. It feeds raw pipelined HTTP bytes through a real HttpServerCodec, exactly as they would arrive from a socket, with the application consuming each request and writing nothing back. Decoded HTTP bytes alone drive the queue to 50,000 entries.

Impact

A remote, unauthenticated attacker holding one TCP connection open can force the server to retain roughly 100 bytes of heap per pipelined request, for the lifetime of that connection, with no upper limit. Memory is reclaimed only when the connection closes, so an attacker who keeps connections open and keeps pipelining drives the server to OutOfMemoryError. Spreading the same traffic across many connections multiplies the effect and keeps any single connection from looking anomalous.

This is unbounded accumulation rather than an amplification bomb. The per-request heap cost is slightly below the per-request wire cost, so the attacker spends bandwidth roughly in proportion to the memory consumed. What makes it a denial of service is that the accumulation has no ceiling and is never released while the connection lives, so the attacker converts

[truncated]

Affected: - maven:io.netty:netty-codec-http affected >= 4.1.88.Final, <= 4.1.137.Final; fixed unknown - maven:io.netty:netty-codec-http affected >= 4.2.0.Final, <= 4.2.17.Final; fixed unknown

Fixed versions: see advisory

Advisory: https://github.com/netty/netty/security/advisories/GHSA-2g37-3h88-55hc