CVE-2022-21698: Uncontrolled Resource Consumption in promhttp

A malicious actor can in theory kill / DOS a server in Go instrumented using prometheus/clientgolang InstrumentHandlerCounter in the version below 1.11.1.

InstrumentHandlerCounter function code: https://github.com/prometheus/clientgolang/blob/22da9497b8f0d53072dfc4721904faa7395d8318/prometheus/promhttp/instrumentserver.go#L95

Security advisory: https://github.com/prometheus/clientgolang/security/advisories/GHSA-cg3q-j54f-5p7p

Upstream fix: https://github.com/prometheus/clientgolang/commit/9075cdf61646b5adf54d3ba77a0e4f6c65cb4fd7 [main] https://github.com/prometheus/clientgolang/commit/989baa30fe956631907493ccee1f8e7708660d96 [release-1.11]

— Red Hat

clientgolang is the instrumentation library for Go applications in Prometheus, and the promhttp package in clientgolang provides tooling around HTTP servers and clients. In clientgolang prior to version 1.11.1, HTTP server is susceptible to a Denial of Service through unbounded cardinality, and potential memory exhaustion, when handling requests with non-standard HTTP methods. In order to be affected, an instrumented software must use any of promhttp.InstrumentHandler middleware except RequestsInFlight; not filter any specific methods (e.g GET) before middleware; pass metric with method label name to our middleware; and not have any firewall/LB/proxy that filters away requests with unknown method. clientgolang version 1.11.1 contains a patch for this issue. Several workarounds are available, including removing the method label name from counter/gauge used in the InstrumentHandler; turning off affected promhttp handlers; adding custom middleware before promhttp handler that will sanitize the request method given by Go http.Request; and using a reverse proxy or web application firewall, configured to only allow a limited set of methods.

This is the Go client library for Prometheus. It has two separate parts, one for instrumenting application code, and one for creating clients that talk to the Prometheus HTTP API. clientgolang is the instrumentation library for Go applications in Prometheus, and the promhttp package in clientgolang provides tooling around HTTP servers and clients.

Impact

HTTP server susceptible to a Denial of Service through unbounded cardinality, and potential memory exhaustion, when handling requests with non-standard HTTP methods.

Affected Configuration

In order to be affected, an instrumented software must

Use any of promhttp.InstrumentHandler middleware except RequestsInFlight. Do not filter any specific methods (e.g GET) before middleware. Pass metric with method label name to our middleware. Not have any firewall/LB/proxy that filters away requests with unknown method.

Patches

https://github.com/prometheus/clientgolang/pull/962 https://github.com/prometheus/clientgolang/pull/987

Workarounds

If you cannot upgrade to v1.11.1 or above, in order to stop being affected you can:

Remove method label name from counter/gauge you use in the InstrumentHandler. Turn off affected promhttp handlers. Add custom middleware before promhttp handler that will sanitize the request method given by Go http.Request. Use a reverse proxy or web application firewall, configured to only allow a limited set of methods.

For more information

If you have any questions or comments about this advisory:

Open an issue in https://github.com/prometheus/clientgolang Email us at prometheus-team@googlegroups.com

— GitHub

Uncontrolled Resource Consumption in promhttp

— Microsoft