GHSA-vmm3-xgcx-67hm: Maven/org.http4s:http4s-ember-core_2.13 vulnerability
Summary
http4s 0.23.x and 1.0 servers running ember with http2 enabled are vulnerable to a denial of service attack using a HPACK bomb vulnerability recently disclosed as affecting other http2 servers.
Impact
Denial of Service: - Affects any http4s server running the ember backend with HTTP/2 enabled that is exposed to untrusted traffic. - Affects any http4s client running the ember backend with HTTP/2 enabled that can be directed to an untrusted server.
Details
The issue occurs in the Hpack wrapper. Ember concatenates the header and continuation frames before decoding all headers at once. When the code decodes the attack payload, the relatively small packets decode to a significantly larger amount of data which is returned as a single List which is then held in memory for further processing. With enough concurrent connections (~5 with a 2GB heap in testing) this leads to an OOM, eg:
java.lang.OutOfMemoryError: Java heap space at scala.collection.mutable.ListBuffer.scala$collection$mutable$ListBuffer$$freshFrom(ListBuffer.scala:129) at scala.collection.mutable.ListBuffer.addAll(ListBuffer.scala:147) at scala.collection.mutable.ListBuffer.addAll(ListBuffer.scala:40) at scala.collection.mutable.Growable.$plus$plus$eq(Growable.scala:69) at scala.collection.mutable.Growable.$plus$plus$eq$(Growable.scala:69) at scala.collection.mutable.AbstractBuffer.$plus$plus$eq(Buffer.scala:314) at org.http4s.Header$.org$http4s$Header$ToRaw$$anon$10$$$$lessinit$greater$$anonfun$2(Header.scala:192) at org.http4s.Header$ToRaw$$anon$10$$Lambda/0x00001fe001239c50.apply(Unknown Source) at scala.collection.immutable.List.foreach(List.scala:334) at org.http4s.Header$ToRaw$$anon$10.<init>(Header.scala:192) at org.http4s.Header$ToRaw$.scalaCollectionSeqToRaw(Header.scala:195) at org.http4s.Headers$.apply(Headers.scala:220) at org.http4s.ember.core.h2.PseudoHeaders$.headersToRequestNoBody(PseudoHeaders.scala:95) at org.http4s.ember.core.h2.H2Stream.receiveHeaders$$anonfun$1$$anonfun$2$$anonfun$3$$anonfun$2(H2Stream.scala:248) at org.http4s.ember.core.h2.H2Stream$$Lambda/0x00001fe001511fd8.apply(Unknown Source) at cats.effect.IOFiber.runLoop(IOFiber.scala:429) at cats.effect.IOFiber.autoCedeR(IOFiber.scala:1460) at cats.effect.IOFiber.run(IOFiber.scala:129) at cats.effect.unsafe.WorkerThread.run(WorkerThread.scala:935)
Fix The Hpack handling code has a configurable max header size parameter, but it does not include indexed headers in that accounting, allowing the attack to bypass the limit. There is not any configuration available in current http4s versions that can be enabled to prevent this attack short of disabling http2.
A fix has been tested locally that threads the maxHeaderSize setting from the server builder into the Hpack code. Using this, connections can be prematurely terminated once the decoded data size exceeds the maxHeaderSize that users can already configure.
Workarounds
If you can't upgrade immediately: - Disable HTTP/2 in Ember backends
Affected Software
Remediation
Recommended actions to resolve this vulnerability, in priority order.
- Upgrade
Upgrade
maven/org.http4s:http4s-ember-core_2.13to a version that resolves this vulnerability.Fixed in 0.23.35 - Upgrade
Upgrade
maven/org.http4s:http4s-ember-core_3to a version that resolves this vulnerability.Fixed in 1.0.0-M47 - Upgrade
Upgrade
maven/org.http4s:http4s-ember-core_3to a version that resolves this vulnerability.Fixed in 0.23.35 - Upgrade
Upgrade
maven/org.http4s:http4s-ember-core_2.13to a version that resolves this vulnerability.Fixed in 1.0.0-M47 - Upgrade
Upgrade
maven/org.http4s:http4s-ember-core_2.12to a version that resolves this vulnerability.Fixed in 0.23.35 - Configuration
Disable HTTP/2 in Ember backends, since there is no configuration in current http4s versions to prevent the HPACK bomb DoS short of disabling http2.
Ember backends (http4s ember-core) HTTP/2 enabled = disabled
Event History
Frequently Asked Questions
Which deployments are exposed to this denial-of-service condition?
Any http4s server using the ember backend with HTTP/2 enabled and exposed to untrusted traffic is affected. http4s clients using ember with HTTP/2 enabled are also affected if they can be directed to an untrusted server.
What does an attacker need to do to trigger the issue?
An attacker needs to send a malicious HTTP/2 HPACK payload through an affected ember HTTP/2 connection. The payload expands from relatively small packets into a much larger in-memory header representation during decoding.
What is the expected operational impact?
The expanded headers are held in memory for further processing, allowing concurrent malicious connections to exhaust the Java heap and cause an OutOfMemoryError. Testing reported heap exhaustion with approximately five concurrent connections on a 2 GB heap.
How can I determine whether an application is in scope?
Check whether the application uses either listed http4s ember-core artifact, runs the ember backend, and has HTTP/2 enabled. For servers, also determine whether untrusted clients can connect; for clients, determine whether an untrusted server can be selected or reached.