GHSA-rcjv-mgp8-qvmr CVE-2023-45142 GO-2023-2113
OpenTelemetry-Go Contrib vulnerable to denial of service in otelhttp due to unbound cardinality metrics
- Severity
- 7.5 / 10
- High · CVSS 3.1
- Exploit chance (EPSS)
- 1.61%
- Chance of exploitation in the next 30 days. Higher than 75.99% of scored vulnerabilities.
- Known exploited
- No
- Not in CISA's catalog
- Weakness
- CWE-770 Allocation of Resources Without Limits or Throttling
- Fix
- Available
- 7 packages have a fixed version
Description
Summary
This handler wrapper https://github.com/open-telemetry/opentelemetry-go-contrib/blob/5f7e6ad5a49b45df45f61a1deb29d7f1158032df/instrumentation/net/http/otelhttp/handler.go#L63-L65 out of the box adds labels
http.user_agenthttp.method
that have unbound cardinality. It leads to the server's potential memory exhaustion when many malicious requests are sent to it.
Details
HTTP header User-Agent or HTTP method for requests can be easily set by an attacker to be random and long. The library internally uses httpconv.ServerRequest that records every value for HTTP method and User-Agent.
PoC
Send many requests with long randomly generated HTTP methods or/and User agents (e.g. a million) and observe how memory consumption increases during it.
Impact
In order to be affected, the program has to configure a metrics pipeline, use otelhttp.NewHandler wrapper, and does not filter any unknown HTTP methods or User agents on the level of CDN, LB, previous middleware, etc.
Others
It is similar to already reported vulnerabilities
- https://github.com/open-telemetry/opentelemetry-go-contrib/security/advisories/GHSA-5r5m-65gx-7vrh (open-telemetry/opentelemetry-go-contrib)
- https://github.com/advisories/GHSA-cg3q-j54f-5p7p (prometheus/client_golang)
Workaround for affected versions
As a workaround to stop being affected otelhttp.WithFilter() can be used, but it requires manual careful configuration to not log certain requests entirely.
For convenience and safe usage of this library, it should by default mark with the label unknown non-standard HTTP methods and User agents to show that such requests were made but do not increase cardinality. In case someone wants to stay with the current behavior, library API should allow to enable it.
The other possibility is to disable HTTP metrics instrumentation by passing otelhttp.WithMeterProvider option with noop.NewMeterProvider.
Solution provided by upgrading
In PR https://github.com/open-telemetry/opentelemetry-go-contrib/pull/4277, released with package version 0.44.0, the values collected for attribute http.request.method were changed to be restricted to a set of well-known values and other high cardinality attributes were removed.
References
Affected versions
Go
| Package | Vulnerable | Fixed in |
|---|---|---|
| go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful | < 0.44.0 |
0.44.0 |
| go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin | < 0.44.0 |
0.44.0 |
| go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux | < 0.44.0 |
0.44.0 |
| go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho | < 0.44.0 |
0.44.0 |
| go.opentelemetry.io/contrib/instrumentation/gopkg.in/macaron.v1/otelmacaron | < 0.44.0 |
0.44.0 |
| go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace | < 0.44.0 |
0.44.0 |
| go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp | < 0.44.0 |
0.44.0 |
How to fix
Move to the fixed version or later with your package manager, update the lockfile, and run your tests. If the package only arrives through another dependency, upgrade that dependency first. If none of its releases pulls a fixed version yet, force the version with your package manager's override.
go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful Go → 0.44.0
- go
go get go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful@v0.44.0 && go mod tidy
go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin Go → 0.44.0
- go
go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin@v0.44.0 && go mod tidy
go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux Go → 0.44.0
- go
go get go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux@v0.44.0 && go mod tidy
go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho Go → 0.44.0
- go
go get go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho@v0.44.0 && go mod tidy
go.opentelemetry.io/contrib/instrumentation/gopkg.in/macaron.v1/otelmacaron Go → 0.44.0
- go
go get go.opentelemetry.io/contrib/instrumentation/gopkg.in/macaron.v1/otelmacaron@v0.44.0 && go mod tidy
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace Go → 0.44.0
- go
go get go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace@v0.44.0 && go mod tidy
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp Go → 0.44.0
- go
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.44.0 && go mod tidy
Severity breakdown
CVSS 3.1 7.5
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
- Attack vector
- Network
- Attack complexity
- Low
- Privileges required
- None
- User interaction
- None
- Scope
- Unchanged
- Confidentiality
- None
- Integrity
- None
- Availability
- High
Weakness
CWE-770: Allocation of Resources Without Limits or Throttling
The product allocates a reusable resource or group of resources on behalf of an actor without imposing any intended restrictions on the size or number of resources that can be allocated.
How to prevent it
- Requirements: Clearly specify the minimum and maximum expectations for capabilities, and dictate which behaviors are acceptable when resource allocation reaches limits.
- Architecture and Design: Limit the amount of resources that are accessible to unprivileged users. Set per-user limits for resources. Allow the system administrator to define these limits. Be careful to avoid CWE-410.
- Architecture and Design: Design throttling mechanisms into the system architecture. The best protection is to limit the amount of resources that an unauthorized user can cause to be expended. A strong authentication and access control model will help prevent such attacks from occurring in the first place, and it will help the administrator to identify who is committing the abuse. The login application should be protected against DoS attacks as much as possible. Limiting the database access, perhaps by caching result sets, can help minimize the resources expended. To further limit the potential for a DoS attack, consider tracking the rate of requests received from users and blocking requests that exceed a defined rate threshold.
- Implementation: Assume all input is malicious. Use an "accept known good" input validation strategy, i.e., use a list of acceptable inputs that strictly conform to specifications. Reject any input that does not strictly conform to specifications, or transform it into something that does. When performing input validation, consider all potentially relevant properties, including length, type of input, the full range of acceptable values, missing or extra inputs, syntax, consistency across related fields, and conformance to business rules. As an example of business rule logic, "boat" may be syntactically valid because it only contains alphanumeric characters, but it is not valid if the input is only expected to contain colors such as "red" or "blue." Do not rely exclusively on looking for malicious or malformed inputs. This is likely to miss at least one undesirable input, especially if the code's environment changes. This can give attackers enough room to bypass the intended validation. However, denylists can be useful for detecting potential attacks or determining which inputs are so malformed that they should be rejected outright.
- Architecture and Design: For any security checks that are performed on the client side, ensure that these checks are duplicated on the server side, in order to avoid CWE-602. Attackers can bypass the client-side checks by modifying values after the checks have been performed, or by changing the client to remove the client-side checks entirely. Then, these modified values would be submitted to the server.
- Architecture and Design: Mitigation of resource exhaustion attacks requires that the target system either: recognizes the attack and denies that user further access for a given amount of time, typically by using increasing time delays uniformly throttles all requests in order to make it more difficult to consume resources more quickly than they can again be freed. The first of these solutions is an issue in itself though, since it may allow attackers to prevent the use of the system by a particular valid user. If the attacker impersonates the valid user, they may be able to prevent the user from accessing the server in question. The second solution can be difficult to effectively institute -- and even when properly done, it does not provide a full solution. It simply requires more resources on the part of the attacker.
- Architecture and Design: Ensure that protocols have specific limits of scale placed on them.
- Architecture and Design, Implementation: If the program must fail, ensure that it fails gracefully (fails closed). There may be a temptation to simply let the program fail poorly in cases such as low memory conditions, but an attacker may be able to assert control before the software has fully exited. Alternately, an uncontrolled failure could cause cascading problems with other downstream components; for example, the program could send a signal to a downstream process so the process immediately knows that a problem has occurred and has a better chance of recovery. Ensure that all failures in resource allocation place the system into a safe posture.
- Operation, Architecture and Design: Use quotas or other resource-limiting settings provided by the operating system or environment. For example, when managing system resources in POSIX, setrlimit() can be used to set limits for certain types of resources, and getrlimit() can determine how many resources are available. However, these functions are not available on all operating systems. When the current levels get close to the maximum that is defined for the application (see CWE-770), then limit the allocation of further resources to privileged users; alternately, begin releasing resources for less-privileged users. While this mitigation may protect the system from attack, it will not necessarily stop attackers from adversely impacting other users. Ensure that the application performs the appropriate error checks and error handling in case resources become unavailable (CWE-703).
References
Advisories
Packages
Other links
- github.com/open-telemetry/opentelemetry-go-contrib/security/advisories/GHSA-5r5…
- github.com/open-telemetry/opentelemetry-go-contrib/security/advisories/GHSA-rcj…
- github.com/open-telemetry/opentelemetry-go-contrib/pull/4277
- github.com/open-telemetry/opentelemetry-go-contrib/blob/5f7e6ad5a49b45df45f61a1…
- github.com/open-telemetry/opentelemetry-go-contrib/releases/tag/v1.19.0
- github.com/open-telemetry/opentelemetry-go/blob/38e1b499c3da3107694ad2660b3888e…
- github.com/open-telemetry/opentelemetry-go/blob/v1.12.0/semconv/internal/v2/htt…
- lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/2UTRJ…
- lists.fedoraproject.org/archives/list/package-announce@lists.fedoraproject.org/message/2UTRJ…
Timeline
- Published
- Last updated by the source
- Latest EPSS score
Related advisories
| Advisory | Severity | Why it's related |
|---|---|---|
| GHSA-5r5m-65gx-7vrh otelhttp and otelbeego have DoS vulnerability for high cardinality metrics | High | Linked by the source |
| GHSA-cg3q-j54f-5p7p Uncontrolled Resource Consumption in promhttp | High | Linked by the source |
| GHSA-62m5-6wh8-28x7 pypdf has possible long runtimes/large memory usage for encrypted PDF files | High | Same weakness, CWE-770 |
| GHSA-9wfm-jcj8-jfv9 russh's SSH agent server is susceptible to unbound allocations via repeated SSH_AGENT_CONSTRAIN_CONFIRM messages | Low | Same weakness, CWE-770 |
| GHSA-4p9f-7p9m-5vfg russh's SSH client is susceptible to unbounded allocations via EXT_INFO packets | Medium | Same weakness, CWE-770 |
| GHSA-x3rh-gjgm-fmqc pypdf has possible large memory usage when parsing cross-reference streams | High | Same weakness, CWE-770 |
| GHSA-8397-8qf6-f2gm vLLM: Phi-4 multimodal audio encoder materializes and retains full attention matrices: a single max-length audio OOM-crashes the vLLM engine | Medium | Same weakness, CWE-770 |
| EEF-CVE-2026-116078 HTTP/2 request bodies made of 1-byte DATA frames amplify memory use in Bandit | Not rated yet | Same weakness, CWE-770 |
MergeHarbor checks every repository's dependencies against this database, ranks what matters by exploitability and reachability, routes it to the owning team and opens the fix pull request.
Check your repos with MergeHarbor