Summary
When exporting telemetry over gRPC using the OpenTelemetry Protocol (OTLP), the exporter may parse a server-provided grpc-status-details-bin trailer during retry handling. Prior to the fix, a malformed trailer could encode an extremely large length-delimited protobuf field which was used directly for allocation, allowing excessive memory allocation and potential denial of service (DoS).
Details
#5980 introduced a retry path that parses grpc-status-details-bin to extract gRPC retry delay information for retryable responses.
On that path:
- OtlpGrpcExportClient captures grpc-status-details-bin from retryable status responses (ResourceExhausted / Unavailable). - OtlpRetry invokes GrpcStatusDeserializer.TryGetGrpcRetryDelay using this untrusted trailer value. - GrpcStatusDeserializer.DecodeBytes decoded a protobuf varint length and allocated new byte[length] without validating the bounds against the remaining payload size.
A malicious or compromised collector (or a MitM in weakly-protected deployments) could return a crafted grpc-status-details-bin payload that forces oversized allocation and memory exhaustion in the instrumented process.
Impact
If an OTLP/gRPC endpoint is attacker-controlled (or traffic is intercepted), a crafted retryable response can trigger large allocations during trailer parsing, which may exhaust memory and cause process instability/crash (availability impact / DoS).
Mitigation
The application's configured back-end/collector endpoint needs to behave maliciously. If the collector/back-end is a well-behaved implementation response bodies should not be excessively large if a request error occurs.
Workarounds
None known.
Remediation
#7064 updates GrpcStatusDeserializer to validate decoded length-delimited field sizes before allocation by ensuring the requested length is sane and does not exceed the remaining payload.
This causes malformed or truncated grpc-status-details-bin payloads to fail safely instead of attempting unbounded allocation.
Summary
When exporting telemetry to a back-end/collector over gRPC or HTTP using OpenTelemetry Protocol format (OTLP), if the request results in a unsuccessful request (i.e. HTTP 4xx or 5xx), the response is read into memory with no upper-bound on the number of bytes consumed.
This could cause memory exhaustion in the consuming application if the configured back-end/collector endpoint is attacker-controlled (or a network attacker can MitM the connection) and an extremely large body is returned by the response.
Details
https://github.com/open-telemetry/opentelemetry-dotnet/pull/6564 introduced a change to read the response body when a non-200 HTTP status code is received when exporting telemetry to aid debugging by operators so that the error response is included in the logs emitted by the exporter for both gRPC and HTTP/protobuf.
An unintended consequence of this change is that the response body is fully read into memory when received with no upper-bound.
This vulnerability was surfaced during the investigation of GHSA-w8rr-5gcm-pp58.
Impact
If an application using the OTLP exporter is configured to use a back-end/collector endpoint that is attacker-controlled (or a network attacker can MitM the connection) and an extremely large body is returned by the response the application could have its memory exhausted and create a denial-of-service condition.
Mitigation
The application's configured back-end/collector endpoint needs to behave maliciously. If the collector/back-end is a well-behaved implementation response bodies should not be excessively large if a request error occurs.
Workarounds
None known.
Remediation
#7017 updates the OTLP exporter for both gRPC and HTTP to:
- Limit the number of bytes read from the response body in an error condition to 4MiB (see https://github.com/open-telemetry/opentelemetry-proto/pull/781); - Only attempt to read the response body if OpenTelemetry error logging is enabled.