CVE-2023-45142: OpenTelemetry-Go Contrib has DoS vulnerability in otelhttp due to unbound cardinality metrics
Summary
This handler wrapper https://github.com/open-telemetry/opentelemetry-go-contrib/blob/5f7e6ad5a49b45df45f61a1deb29d7f1158032df/instrumentation/net/http/otelhttp/handler.go#L63-L65 out of the box adds labels
- http.useragent - http.method
that have unbound cardinality. It leads to the server's potential memory exhaustion when many malicious requests are sent to it.
Details
HTTP header User-Agent or HTTP method for requests can be easily set by an attacker to be random and long. The library internally uses httpconv.ServerRequest that records every value for HTTP method and User-Agent.
PoC
Send many requests with long randomly generated HTTP methods or/and User agents (e.g. a million) and observe how memory consumption increases during it.
Impact
In order to be affected, the program has to configure a metrics pipeline, use otelhttp.NewHandler wrapper, and does not filter any unknown HTTP methods or User agents on the level of CDN, LB, previous middleware, etc.
Others
It is similar to already reported vulnerabilities - https://github.com/open-telemetry/opentelemetry-go-contrib/security/advisories/GHSA-5r5m-65gx-7vrh (open-telemetry/opentelemetry-go-contrib) - https://github.com/advisories/GHSA-cg3q-j54f-5p7p (prometheus/clientgolang)
Workaround for affected versions
As a workaround to stop being affected otelhttp.WithFilter() can be used, but it requires manual careful configuration to not log certain requests entirely.
For convenience and safe usage of this library, it should by default mark with the label unknown non-standard HTTP methods and User agents to show that such requests were made but do not increase cardinality. In case someone wants to stay with the current behavior, library API should allow to enable it.
The other possibility is to disable HTTP metrics instrumentation by passing otelhttp.WithMeterProvider option with noop.NewMeterProvider.
Solution provided by upgrading
In PR https://github.com/open-telemetry/opentelemetry-go-contrib/pull/4277, released with package version 0.44.0, the values collected for attribute http.request.method were changed to be restricted to a set of well-known values and other high cardinality attributes were removed.
References
- https://github.com/open-telemetry/opentelemetry-go-contrib/pull/4277 - https://github.com/open-telemetry/opentelemetry-go-contrib/releases/tag/v1.19.0
Other sources
OpenTelemetry-Go Contrib has DoS vulnerability in otelhttp due to unbound cardinality metrics
— Microsoft
OpenTelemetry-Go Contrib is a collection of third-party packages for OpenTelemetry-Go. A handler wrapper out of the box adds labels http.useragent and http.method that have unbound cardinality. It leads to the server's potential memory exhaustion when many malicious requests are sent to it. HTTP header User-Agent or HTTP method for requests can be easily set by an attacker to be random and long. The library internally uses httpconv.ServerRequest that records every value for HTTP method and User-Agent. In order to be affected, a program has to use the otelhttp.NewHandler wrapper and not filter any unknown HTTP methods or User agents on the level of CDN, LB, previous middleware, etc. Version 0.44.0 fixed this issue when the values collected for attribute http.request.method were changed to be restricted to a set of well-known values and other high cardinality attributes were removed. As a workaround to stop being affected, otelhttp.WithFilter() can be used, but it requires manual careful configuration to not log certain requests entirely. For convenience and safe usage of this library, it should by default mark with the label unknown non-standard HTTP methods and User agents to show that such requests were made but do not increase cardinality. In case someone wants to stay with the current behavior, library API should allow to enable it.
— MITRE
Affected Software
Remediation
Recommended actions to resolve this vulnerability, in priority order.
- Upgrade
Upgrade
go/go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptraceto a version that resolves this vulnerability.Fixed in 0.44.0 - Upgrade
Upgrade
go/go.opentelemetry.io/contrib/instrumentation/gopkg.in/macaron.v1/otelmacaronto a version that resolves this vulnerability.Fixed in 0.44.0 - Upgrade
Upgrade
go/go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelechoto a version that resolves this vulnerability.Fixed in 0.44.0 - Upgrade
Upgrade
go/go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmuxto a version that resolves this vulnerability.Fixed in 0.44.0 - Upgrade
Upgrade
go/go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelginto a version that resolves this vulnerability.Fixed in 0.44.0 - Upgrade
Upgrade
go/go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestfulto a version that resolves this vulnerability.Fixed in 0.44.0 - Upgrade
Upgrade
go/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttpto a version that resolves this vulnerability.Fixed in 0.44.0 - Upgrade
Upgrade
redhat/opentelemetry-goto a version that resolves this vulnerability.Fixed in 0.44.0 - Configuration
Use otelhttp.WithFilter() to prevent recording metrics for requests with untrusted or high-cardinality HTTP methods and User-Agent headers; carefully configure the filter to avoid dropping legitimate traffic entirely.
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp WithFilter = apply filter to exclude or ignore requests with non-standard/high-cardinality http.method and http.user_agent values - Configuration
Disable HTTP metrics instrumentation by passing otelhttp.WithMeterProvider(noop.NewMeterProvider()) so no high-cardinality HTTP metrics are produced.
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp WithMeterProvider = noop.NewMeterProvider - Compensating control
Filter or normalize HTTP methods and User-Agent headers at the CDN, load balancer, WAF, or ingress (e.g., block or map non-standard/random methods and long/unknown User-Agent values, enforce rate limits) to prevent unbounded metric cardinality prior to the application.
Event History
Frequently Asked Questions
What is CVE-2023-45142?
CVE-2023-45142 is a vulnerability in OpenTelemetry-Go Contrib that has a high severity rating of 7.5.
What is the impact of CVE-2023-45142?
CVE-2023-45142 allows an attacker to cause a denial-of-service (DoS) or potentially execute arbitrary code.
Which versions of OpenTelemetry-Go Contrib are affected by CVE-2023-45142?
OpenTelemetry-Go Contrib versions up to (but not including) 0.44.0 are affected by CVE-2023-45142.
How do I fix CVE-2023-45142?
To fix CVE-2023-45142, update to OpenTelemetry-Go Contrib version 0.44.0 or later.
Where can I find more information about CVE-2023-45142?
You can find more information about CVE-2023-45142 in the GitHub security advisory and the OpenTelemetry-Go Contrib GitHub repository.